-----============= acceptance-small: sanity-flr ============----- Mon Jun 1 13:28:10 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg143-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 13:28:33 (1780334913) === oleg143-client.virtnet: executing check_config_client /mnt/lustre oleg143-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg143-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98529100b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98529100b000.idle_timeout=debug disable quota as required oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 13:29:18 (1780334958) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8189] == sanity-flr test 0a: lfs mirror create with -N option == 13:29:21 (1780334961) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:29:41 (1780334981) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:29:45 (1780334985) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:29:49 (1780334989) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:30:19 (1780335019) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:30:24 (1780335024) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 13:30:28 (1780335028) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff98529100b000.stats=clear osc.lustre-OST0001-osc-ffff98529100b000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:30:37 (1780335037) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:30:47 (1780335047) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:30:57 (1780335057) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:31:01 (1780335061) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:31:11 (1780335071) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 13:31:22 (1780335082) PASS 2 (11s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:31:33 (1780335093) PASS 3 (11s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:31:44 (1780335104) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:31:53 (1780335113) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (10s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:32:07 (1780335127) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:32:15 (1780335135) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.354329 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13147 s, 8.0 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:32:25 (1780335145) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548168 s, 19.1 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:32:32 (1780335152) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:32:42 (1780335162) Stopping /mnt/lustre-ost1 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98529100b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98529100b000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg143-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg143-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg143-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg143-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg143-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg143-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg143-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg143-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg143-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg143-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg143-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg143-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg143-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg143-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg143-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg143-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg143-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg143-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg143-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg143-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg143-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg143-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg143-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg143-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg143-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98529100b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98529100b000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg143-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (137s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:34:59 (1780335299) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.77292 s, 14.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98529100b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98529100b000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg143-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:35:44 (1780335344) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98529100b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98529100b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg143-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98529100b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98529100b000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg143-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:37:00 (1780335420) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.36517 s, 15.6 MB/s -rw-r--r-- 1 root root 52428800 Jun 1 13:37 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.219s user 0m0.001s sys 0m0.168s Stopping /mnt/lustre-ost1 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98529100b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98529100b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.128s user 0m0.000s sys 0m0.179s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg143-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98529100b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98529100b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.818s user 0m0.000s sys 0m0.187s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg143-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg143-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98529100b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98529100b000.ost_server_uuid in FULL state after 0 sec PASS 33b (84s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:38:24 (1780335504) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:38:27 (1780335507) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:38:30 (1780335510) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 13:38:33 (1780335513) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 13:38:40 (1780335520) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.712331 s, 17.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:38:53 (1780335533) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031799 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037553 s, 27.9 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:39:00 (1780335540) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:39:08 (1780335548) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000714637 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000874936 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0023413 s, 437 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00116251 s, 881 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 13:39:18 (1780335558) create test files with size 15735294 .. cd8cb7b516ed853ecd89f61c993ebdc8 - 65d74d805f889012781bc3e240c90f3e - 922367d8322b83de7d7e996243b0a9fe - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg143-client.virtnet /mnt/lustre (opts:) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 13:39:43 (1780335583) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:39:56 (1780335596) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:40:01 (1780335601) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10524 s, 10.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584182 s, 17.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 13:40:12 (1780335612) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0974335 s, 21.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.243638 s, 17.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20736 s, 10.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 13:40:30 (1780335630) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 13:40:34 (1780335634) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:40:39 (1780335639) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623001 s, 16.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 13:40:47 (1780335647) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.429595 s, 7.3 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:40:58 (1780335658) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:41:11 (1780335671) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.324648 s, 32.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg143-client.virtnet /mnt/lustre (opts:) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:41:21 (1780335681) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff98529165f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.257614 s, 40.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg143-client.virtnet /mnt/lustre (opts:) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9852b152e000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:41:36 (1780335696) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:42:20 (1780335740) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:42:58 (1780335778) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 13:43:02 (1780335782) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.339672 s, 9.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.172321 s, 18.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xce237148 (nosync,stale) Mirror 2: 0xce237148 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 9d58171bdc4c8520212c625549fa8832 - mirror 1 checksum: 18a8a4712933920d97a793b9e0874239 - mirror 2 checksum: 9d58171bdc4c8520212c625549fa8832 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 13:43:04 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 13:43:04 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9d58171bdc4c8520212c625549fa8832 - mirror 1 checksum: 18a8a4712933920d97a793b9e0874239 - mirror 2 checksum: 18a8a4712933920d97a793b9e0874239 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 13:43:04 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 13:43:04 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 13:43:13 (1780335793) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325784 s, 32.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:43:21 (1780335801) Failing mds1 on oleg143-server Stopping /mnt/lustre-mds1 (opts:) on oleg143-server 13:43:33 (1780335813) shut down facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server Failover mds1 to oleg143-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 13:43:55 (1780335835) targets are mounted 13:43:55 (1780335835) facet_failover done oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247302 s, 42.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 50A (51s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:44:12 (1780335852) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.057755 s, 355 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0117754 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:44:23 (1780335863) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0729437 s, 281 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0213685 s, 958 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:44:33 (1780335873) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0156172 s, 1.0 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:44:42 (1780335882) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0474417 s, 432 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:44:51 (1780335891) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0252269 s, 162 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.03776 s, 108 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:44:59 (1780335899) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:45:22 (1780335922) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:45:52 (1780335952) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:46:16 (1780335976) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:46:41 (1780336001) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 13:46:48 (1780336008) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57151 mirror split pid 57152 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57151 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57152 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 13:47:57 (1780336077) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 6079 000001 1780336079.690001 mapwrite 0x022076 thru 0x022cf6 (0xc81 bytes) truncating to largest ever: 0x11079 000002 1780336079.740790 trunc from 0x022cf7 to 0x011078 (0x11c7e bytes) truncating to largest ever: 0x37e38 000003 1780336079.755323 trunc from 0x011079 to 0x037e37 (0x26dbf bytes) 000004 1780336079.769172 trunc from 0x037e38 to 0x001c68 (0x361cf bytes) 000005 1780336079.786914 read 0x000bb1 thru 0x001c68 (0x10b8 bytes) 000006 1780336079.793806 trunc from 0x001c69 to 0x01e4d7 (0x1c86f bytes) 000007 1780336079.830518 MIRROR_EXTEND to 2 mirrors 000008 1780336080.543905 punch from 0x01321c to 0x016eae (0x3c93 bytes) 000009 1780336080.601230 write 0x0294cb thru 0x032660 (0x9196 bytes) 000010 1780336080.621313 write 0x032385 thru 0x03fa39 (0xd6b5 bytes) 000011 1780336080.626931 mapread 0x008083 thru 0x00ac8d (0x2c0b bytes) 000012 1780336080.673707 mapread 0x01203d thru 0x01462b (0x25ef bytes) 000013 1780336080.698589 MIRROR_RESYNC 2 mirrors 000014 1780336081.418584 trunc from 0x03e3db to 0x03fa39 (0x165f bytes) 000015 1780336081.468909 read 0x030b65 thru 0x03a0f2 (0x958e bytes) 000016 1780336081.486882 punch from 0x010ac9 to 0x015464 (0x499c bytes) 000017 1780336081.493793 mapread 0x0182f0 thru 0x025866 (0xd577 bytes) 000018 1780336081.552322 fallocate 0x01cabf thru 0x027d41 (0xb283 bytes) 000019 1780336081.559432 write 0x009f06 thru 0x0115a7 (0x76a2 bytes) 000020 1780336081.573293 mapread 0x03b7ae thru 0x03fa39 (0x428c bytes) 000021 1780336081.584701 mapread 0x01d082 thru 0x02bf11 (0xee90 bytes) 000022 1780336081.586345 trunc from 0x03fa3a to 0x00fc9b (0x2fd9e bytes) 000023 1780336081.611733 trunc from 0x00fc9c to 0x01121a (0x157f bytes) 000024 1780336081.634512 trunc from 0x00b945 to 0x01121a (0x58d6 bytes) 000025 1780336081.656685 mapwrite 0x004b3d thru 0x009e9a (0x535e bytes) 000026 1780336081.694065 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1121b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1121b): Mirror 2: 0xcebb44ef (stale) CRC-32 checksum value for chunk [0, 0x1121b): Mirror 1: 0x41afebd7 Mirror 3: 0x41afebd7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1121b: skipped 000027 1780336082.555096 mapwrite 0x028ac3 thru 0x0318e1 (0x8e1f bytes) 000030 1780336082.725561 read 0x0151bf thru 0x018fe1 (0x3e23 bytes) 000031 1780336082.733028 trunc from 0x0318e2 to 0x0004b6 (0x3142b bytes) 000032 1780336082.753388 punch from 0x0004b4 to 0x0004b6 (0x3 bytes) 000033 1780336082.764802 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x4b7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4b7): Mirror 2: 0x8094a6d2 (stale) Mirror 3: 0x8094a6d2 (stale) CRC-32 checksum value for chunk [0, 0x4b7): Mirror 1: 0x8094a6d2 Mirror 4: 0x8094a6d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4b7: skipped fallocating to largest ever: 0x40000 000034 1780336083.616219 fallocate 0x03e434 thru 0x03ffff (0x1bcc bytes) 000035 1780336083.687533 read 0x000342 thru 0x0004b6 (0x175 bytes) 000037 1780336083.696408 punch from 0x0000d9 to 0x0004b6 (0x3de bytes) 000038 1780336083.707168 fallocate 0x00f2c7 thru 0x0156d0 (0x640a bytes) 000039 1780336083.716536 mapread 0x0103a9 thru 0x0156d0 (0x5328 bytes) 000040 1780336083.745172 fallocate 0x00a149 thru 0x0146b3 (0xa56b bytes) 000041 1780336083.752562 punch from 0x001ecc to 0x007310 (0x5445 bytes) 000042 1780336083.757707 mapwrite 0x009aa5 thru 0x00f7d1 (0x5d2d bytes) 000043 1780336083.810213 punch from 0x00ae2e to 0x00e2de (0x34b1 bytes) 000044 1780336083.821616 punch from 0x009348 to 0x00e636 (0x52ef bytes) 000045 1780336083.834530 mapwrite 0x00ef6d thru 0x01ba8a (0xcb1e bytes) 000046 1780336083.923111 write 0x033fb1 thru 0x03a15f (0x61af bytes) 000047 1780336083.924704 trunc from 0x01d94d to 0x02ce17 (0xf4cb bytes) 000049 1780336083.930278 write 0x027b7a thru 0x035e28 (0xe2af bytes) 000050 1780336083.947760 fallocate 0x0244ba thru 0x02b0b2 (0x6bf9 bytes) 000051 1780336083.952713 mapread 0x001835 thru 0x0031f0 (0x19bc bytes) 000052 1780336083.958500 write 0x003b0b thru 0x01307d (0xf573 bytes) 000053 1780336083.960208 fallocate 0x01a51a thru 0x01fda2 (0x5889 bytes) 000055 1780336083.967753 trunc from 0x01459c to 0x016d6f (0x27d4 bytes) 000056 1780336083.979335 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a160): Mirror 2: 0x594a19b4 (stale) Mirror 3: 0x814eafa8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a160): Mirror 2: 0x594a19b4 (stale) Mirror 3: 0x814eafa8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a160: skipped 000057 1780336084.339636 mapwrite 0x03e798 thru 0x03ffff (0x1868 bytes) 000058 1780336084.391275 trunc from 0x040000 to 0x0086dc (0x37923 bytes) 000059 1780336084.411280 mapwrite 0x02bf26 thru 0x038f71 (0xd04c bytes) 000061 1780336084.477115 mapwrite 0x00a297 thru 0x017bba (0xd924 bytes) 000062 1780336084.536257 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38f72): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f72): Mirror 2: 0xa5df418c (stale) Mirror 3: 0xd43037b8 (stale) CRC-32 checksum value for chunk [0, 0x38f72): Mirror 1: 0x3c8814fe Mirror 4: 0x3c8814fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f72: skipped 000063 1780336085.534357 trunc from 0x038f72 to 0x017bcb (0x213a6 bytes) 000064 1780336085.600778 read 0x00fc93 thru 0x010b19 (0xe87 bytes) 000065 1780336085.607537 read 0x00849f thru 0x00a13d (0x1c9f bytes) 000066 1780336085.628597 read 0x0122d4 thru 0x017bcb (0x58f8 bytes) 000067 1780336085.630549 punch from 0x00e27d to 0x017bcb (0x994f bytes) 000068 1780336085.636425 trunc from 0x017bcc to 0x016c64 (0xf67 bytes) 000070 1780336085.663198 mapwrite 0x00142c thru 0x004722 (0x32f7 bytes) 000071 1780336085.680215 write 0x021459 thru 0x02a12c (0x8cd4 bytes) 000072 1780336085.682670 trunc from 0x02a12d to 0x01341a (0x16d12 bytes) 000073 1780336085.709095 punch from 0x0126d1 to 0x01341a (0xd4a bytes) 000074 1780336085.715343 mapwrite 0x02e449 thru 0x037d77 (0x992f bytes) 000075 1780336085.768890 fallocate 0x02c03a thru 0x039f96 (0xdf5d bytes) 000076 1780336085.775772 fallocate 0x0009e3 thru 0x003b22 (0x3140 bytes) 000077 1780336085.781759 trunc from 0x037d78 to 0x02e754 (0x9623 bytes) 000078 1780336085.803101 trunc from 0x01be53 to 0x020ac7 (0x4c75 bytes) 000079 1780336085.809573 mapwrite 0x01980a thru 0x025dd5 (0xc5cc bytes) 000080 1780336085.880214 write 0x0187b6 thru 0x028551 (0xfd9c bytes) 000081 1780336085.886305 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2e755): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e755): Mirror 2: 0xbe150f45 (stale) Mirror 3: 0x9d6932a2 (stale) Mirror 4: 0x1f256dda (stale) CRC-32 checksum value for chunk [0, 0x2e755): Mirror 1: 0xc4b58a7 Mirror 5: 0xc4b58a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e755: skipped 000082 1780336086.694577 punch from 0x001265 to 0x007c90 (0x6a2c bytes) 000083 1780336086.743624 read 0x000107 thru 0x00a4e6 (0xa3e0 bytes) 000084 1780336086.755314 write 0x016ee9 thru 0x019463 (0x257b bytes) 000085 1780336086.777206 fallocate 0x0256f5 thru 0x02dc99 (0x85a5 bytes) 000086 1780336086.783076 fallocate 0x01a1ad thru 0x0283fc (0xe250 bytes) 000087 1780336086.787731 trunc from 0x029904 to 0x02e754 (0x4e51 bytes) 000088 1780336086.793032 trunc from 0x016cb3 to 0x01e6ce (0x7a1c bytes) 000089 1780336086.804706 trunc from 0x01507c to 0x023a38 (0xe9bd bytes) 000091 1780336086.810815 read 0x0238a7 thru 0x02e754 (0xaeae bytes) 000092 1780336086.820582 punch from 0x00e79b to 0x01abe0 (0xc446 bytes) 000093 1780336086.830003 punch from 0x003f44 to 0x008282 (0x433f bytes) 000094 1780336086.837931 trunc from 0x02e755 to 0x008f5f (0x257f5 bytes) 000095 1780336086.884906 punch from 0x004099 to 0x0078f2 (0x385a bytes) 000096 1780336086.894775 trunc from 0x008f60 to 0x01eab5 (0x15b56 bytes) 000097 1780336086.921696 fallocate 0x02de46 thru 0x03d7aa (0xf965 bytes) 000098 1780336086.927644 mapread 0x013306 thru 0x0182ab (0x4fa6 bytes) 000099 1780336086.951057 mapread 0x01a112 thru 0x01eab5 (0x49a4 bytes) 000100 1780336086.984930 mapwrite 0x019043 thru 0x0263cd (0xd38b bytes) 000101 1780336087.162546 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x263ce): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x263ce): Mirror 2: 0x38c27c91 (stale) Mirror 3: 0x8977e24a (stale) Mirror 4: 0xa2feadb0 (stale) Mirror 5: 0x6188d45d (stale) CRC-32 checksum value for chunk [0, 0x263ce): Mirror 1: 0x22d66eb7 Mirror 6: 0x22d66eb7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x263ce: skipped 000102 1780336088.332652 mapread 0x0124d9 thru 0x021c7e (0xf7a6 bytes) 000104 1780336088.475760 mapwrite 0x038e98 thru 0x03b5f7 (0x2760 bytes) 000105 1780336088.667656 write 0x01e85c thru 0x02b641 (0xcde6 bytes) 000106 1780336088.707653 read 0x0269d4 thru 0x035193 (0xe7c0 bytes) 000107 1780336088.721585 mapread 0x02b024 thru 0x038582 (0xd55f bytes) 000108 1780336088.724747 write 0x017243 thru 0x026610 (0xf3ce bytes) 000109 1780336088.727199 write 0x03b1da thru 0x03c9d8 (0x17ff bytes) 000110 1780336088.739678 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c9d9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9d9): Mirror 2: 0x396aba08 (stale) Mirror 3: 0x641b2437 (stale) Mirror 4: 0x46ad4248 (stale) Mirror 5: 0xe8eff82a (stale) Mirror 6: 0x52a74497 (stale) CRC-32 checksum value for chunk [0, 0x3c9d9): Mirror 1: 0x10353a Mirror 7: 0x10353a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9d9: skipped 000111 1780336090.028353 punch from 0x00b0a9 to 0x00e683 (0x35db bytes) 000112 1780336090.131170 mapread 0x000570 thru 0x005884 (0x5315 bytes) 000113 1780336090.183337 fallocate 0x02f2e0 thru 0x039285 (0x9fa6 bytes) 000115 1780336090.191652 trunc from 0x03c9d9 to 0x023918 (0x190c0 bytes) 000116 1780336090.252633 fallocate 0x025c6e thru 0x0262c8 (0x65b bytes) 000117 1780336090.258625 punch from 0x00d399 to 0x0142ae (0x6f16 bytes) 000118 1780336090.264472 punch from 0x026115 to 0x0262c8 (0x1b4 bytes) 000119 1780336090.268760 punch from 0x000272 to 0x00bb7d (0xb90c bytes) 000120 1780336090.277177 fallocate 0x02aa7a thru 0x03497f (0x9f06 bytes) 000121 1780336090.286259 trunc from 0x0262c9 to 0x02139b (0x4f2d bytes) 000124 1780336090.317371 trunc from 0x0196bf to 0x01ebe9 (0x552b bytes) 000125 1780336090.333828 write 0x0199df thru 0x01d9da (0x3ffc bytes) 000126 1780336090.370531 trunc from 0x00b336 to 0x011bb5 (0x6880 bytes) 000127 1780336090.411051 MIRROR_RESYNC 7 mirrors 000128 1780336091.964507 trunc from 0x02139c to 0x010ed2 (0x104c9 bytes) 000129 1780336092.026402 mapwrite 0x00e6ca thru 0x01e1da (0xfb11 bytes) 000130 1780336092.126349 read 0x0027ba thru 0x0126eb (0xff32 bytes) 000131 1780336092.138162 mapread 0x00782c thru 0x0159c3 (0xe198 bytes) 000132 1780336092.140562 trunc from 0x013b78 to 0x01e039 (0xa4c2 bytes) 000133 1780336092.149537 read 0x00a1bd thru 0x00b16c (0xfb0 bytes) 000134 1780336092.151378 punch from 0x00add5 to 0x00b393 (0x5bf bytes) 000135 1780336092.158311 read 0x016a05 thru 0x01b4cb (0x4ac7 bytes) 000136 1780336092.170428 mapwrite 0x03d89e thru 0x03ee3e (0x15a1 bytes) 000137 1780336092.232947 trunc from 0x03b6aa to 0x03ee3e (0x3795 bytes) 000138 1780336092.240571 write 0x015aec thru 0x01e173 (0x8688 bytes) 000139 1780336092.243244 write 0x011322 thru 0x018962 (0x7641 bytes) 000141 1780336092.248429 mapwrite 0x0088e1 thru 0x0132dd (0xa9fd bytes) 000142 1780336092.272991 trunc from 0x013db3 to 0x016175 (0x23c3 bytes) 000143 1780336092.286132 punch from 0x02a105 to 0x02d9c7 (0x38c3 bytes) 000144 1780336092.292493 mapread 0x01dd45 thru 0x0245ff (0x68bb bytes) 000145 1780336092.294690 read 0x030708 thru 0x03e41e (0xdd17 bytes) 000146 1780336092.308140 punch from 0x025ab0 to 0x0284ad (0x29fe bytes) 000147 1780336092.313719 mapwrite 0x02976c thru 0x033bb3 (0xa448 bytes) 000148 1780336092.365379 punch from 0x012d90 to 0x01ba05 (0x8c76 bytes) 000149 1780336092.374055 trunc from 0x03ee3f to 0x025b95 (0x192a9 bytes) 000150 1780336092.392132 trunc from 0x025b96 to 0x017c74 (0xdf21 bytes) 000151 1780336092.419177 trunc from 0x017c75 to 0x02261c (0xa9a8 bytes) 000152 1780336092.447811 fallocate 0x02474d thru 0x02d01b (0x88cf bytes) 000154 1780336092.460881 read 0x02b3dd thru 0x02d01b (0x1c3f bytes) 000155 1780336092.472432 fallocate 0x015622 thru 0x022557 (0xcf36 bytes) 000156 1780336092.478032 write 0x0329b1 thru 0x0352ac (0x28fc bytes) 000157 1780336092.479964 punch from 0x03044b to 0x031fad (0x1b63 bytes) 000158 1780336092.489000 mapread 0x034839 thru 0x0352ac (0xa74 bytes) 000159 1780336092.490330 trunc from 0x03439d to 0x0352ac (0xf10 bytes) 000160 1780336092.505317 trunc from 0x00387a to 0x005f6f (0x26f6 bytes) 000161 1780336092.525207 punch from 0x000832 to 0x008e8c (0x865b bytes) 000162 1780336092.539648 mapwrite 0x02397c thru 0x02b512 (0x7b97 bytes) 000163 1780336092.572248 punch from 0x020155 to 0x02125d (0x1109 bytes) 000164 1780336092.579481 punch from 0x000bc3 to 0x00237a (0x17b8 bytes) 000165 1780336092.586505 read 0x00ddf9 thru 0x01c6ed (0xe8f5 bytes) 000166 1780336092.588434 fallocate 0x01a7d6 thru 0x0255c1 (0xadec bytes) 000167 1780336092.596027 mapwrite 0x02eee0 thru 0x03a9ec (0xbb0d bytes) 000168 1780336092.715548 punch from 0x0366ec to 0x03a59f (0x3eb4 bytes) 000169 1780336092.722460 mapread 0x00c54d thru 0x013f9f (0x7a53 bytes) 000170 1780336092.723534 punch from 0x027908 to 0x0303b7 (0x8ab0 bytes) 000171 1780336092.733632 mapwrite 0x002444 thru 0x004e36 (0x29f3 bytes) 000172 1780336092.749786 mapwrite 0x02ee1f thru 0x03dfc5 (0xf1a7 bytes) 000173 1780336092.866987 trunc from 0x03dfc6 to 0x00914c (0x34e79 bytes) 000174 1780336092.895595 fallocate 0x039e36 thru 0x03ffff (0x61ca bytes) 000175 1780336092.903468 trunc from 0x00914d to 0x006929 (0x2823 bytes) 000176 1780336092.921005 mapread 0x00400a thru 0x006929 (0x2920 bytes) 000177 1780336092.923156 mapread 0x000f99 thru 0x00260d (0x1675 bytes) 000178 1780336092.924917 punch from 0x003d12 to 0x006929 (0x2c18 bytes) 000179 1780336092.930494 read 0x005bde thru 0x006929 (0xd4c bytes) 000180 1780336092.939330 trunc from 0x00692a to 0x018cdc (0x123b3 bytes) 000181 1780336092.961492 trunc from 0x00f656 to 0x010f9d (0x1948 bytes) 000182 1780336092.968141 punch from 0x009d7b to 0x00f4c7 (0x574d bytes) 000183 1780336092.975038 fallocate 0x034ac9 thru 0x03a222 (0x575a bytes) 000184 1780336092.981145 trunc from 0x018cdd to 0x000dd8 (0x17f04 bytes) 000186 1780336093.003657 trunc from 0x000dd9 to 0x02e855 (0x2da7d bytes) 000187 1780336093.034192 trunc from 0x022587 to 0x02e855 (0xc2cf bytes) 000188 1780336093.039653 mapread 0x004b67 thru 0x012e25 (0xe2bf bytes) 000189 1780336093.111968 read 0x02ba83 thru 0x02e855 (0x2dd3 bytes) 000190 1780336093.118701 MIRROR_RESYNC 7 mirrors 000191 1780336094.292943 read 0x008984 thru 0x00bb41 (0x31be bytes) 000192 1780336094.303092 trunc from 0x02e856 to 0x03ce79 (0xe624 bytes) 000193 1780336094.365455 trunc from 0x03ce7a to 0x02186b (0x1b60e bytes) 000194 1780336094.394361 trunc from 0x02186c to 0x01e892 (0x2fd9 bytes) 000195 1780336094.423117 read 0x0028fe thru 0x010212 (0xd915 bytes) 000196 1780336094.446719 read 0x0030ac thru 0x006a3d (0x3992 bytes) 000197 1780336094.451528 write 0x02b80b thru 0x038a25 (0xd21b bytes) 000198 1780336094.462424 mapread 0x010f23 thru 0x01db6a (0xcc48 bytes) 000199 1780336094.467160 trunc from 0x00eadb to 0x01bf9a (0xd4c0 bytes) 000200 1780336094.475729 mapread 0x0359aa thru 0x038a25 (0x307c bytes) 000201 1780336094.476983 punch from 0x010c0b to 0x01c054 (0xb44a bytes) 000202 1780336094.489495 mapwrite 0x0166fb thru 0x01d884 (0x718a bytes) 000204 1780336094.535477 punch from 0x023088 to 0x030e75 (0xddee bytes) 000205 1780336094.543427 fallocate 0x00fdf2 thru 0x018662 (0x8871 bytes) 000207 1780336094.555740 trunc from 0x017dd8 to 0x01de5d (0x6086 bytes) 000208 1780336094.563474 read 0x011421 thru 0x015148 (0x3d28 bytes) 000209 1780336094.565658 fallocate 0x0379ac thru 0x03eb0a (0x715f bytes) 000211 1780336094.572759 read 0x00257a thru 0x004f45 (0x29cc bytes) 000212 1780336094.573476 trunc from 0x038022 to 0x03eb0a (0x6ae9 bytes) 000213 1780336094.581633 punch from 0x0018a7 to 0x00a306 (0x8a60 bytes) 000214 1780336094.587896 trunc from 0x03eb0b to 0x008824 (0x362e6 bytes) 000215 1780336094.613356 trunc from 0x00447a to 0x008824 (0x43ab bytes) 000216 1780336094.629803 mapwrite 0x0396bd thru 0x03b6eb (0x202f bytes) 000217 1780336094.727768 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3b6ec): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6ec): Mirror 2: 0x969bcb24 (stale) Mirror 3: 0x969bcb24 (stale) Mirror 4: 0x969bcb24 (stale) Mirror 5: 0x969bcb24 (stale) Mirror 6: 0x969bcb24 (stale) Mirror 7: 0x969bcb24 (stale) CRC-32 checksum value for chunk [0, 0x3b6ec): Mirror 1: 0x42924eca Mirror 8: 0x42924eca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6ec: skipped 000218 1780336095.831829 trunc from 0x03b6ec to 0x02a465 (0x11286 bytes) 000219 1780336095.896987 mapwrite 0x00c6f4 thru 0x0112ae (0x4bbb bytes) 000220 1780336095.958048 fallocate 0x03a62a thru 0x03ffff (0x59d6 bytes) 000221 1780336095.988792 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a466): Mirror 3: 0xb4928172 (stale) Mirror 4: 0xb4928172 (stale) Mirror 5: 0xb4928172 (stale) Mirror 6: 0xb4928172 (stale) Mirror 7: 0xb4928172 (stale) Mirror 8: 0xb4928172 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a466): Mirror 3: 0xb4928172 (stale) Mirror 4: 0xb4928172 (stale) Mirror 5: 0xb4928172 (stale) Mirror 6: 0xb4928172 (stale) Mirror 7: 0xb4928172 (stale) Mirror 8: 0xb4928172 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a466: skipped 000222 1780336096.606582 mapread 0x011f9b thru 0x0159d5 (0x3a3b bytes) 000223 1780336096.646651 trunc from 0x02a466 to 0x03cb43 (0x126de bytes) 000224 1780336096.688637 write 0x037342 thru 0x03b530 (0x41ef bytes) 000225 1780336096.706506 mapwrite 0x00c95e thru 0x00f5ce (0x2c71 bytes) 000226 1780336096.733563 mapread 0x00b5e2 thru 0x00e50a (0x2f29 bytes) 000227 1780336096.747399 mapread 0x01882e thru 0x01a79f (0x1f72 bytes) 000228 1780336096.749480 trunc from 0x03ba29 to 0x03cb43 (0x111b bytes) 000229 1780336096.757525 mapwrite 0x02e7e4 thru 0x034d1c (0x6539 bytes) 000230 1780336096.792915 read 0x03aaac thru 0x03cb43 (0x2098 bytes) 000232 1780336096.801464 write 0x011d99 thru 0x018f36 (0x719e bytes) 000233 1780336096.803499 mapread 0x021afa thru 0x02383a (0x1d41 bytes) 000234 1780336096.826048 write 0x02f2e9 thru 0x030c62 (0x197a bytes) 000235 1780336096.827125 mapread 0x0273b4 thru 0x02a010 (0x2c5d bytes) 000236 1780336096.829431 punch from 0x015ce3 to 0x0245f0 (0xe90e bytes) 000238 1780336096.856620 trunc from 0x00b02a to 0x01336e (0x8345 bytes) 000239 1780336096.867665 mapwrite 0x0381f6 thru 0x03ef85 (0x6d90 bytes) 000240 1780336096.974452 fallocate 0x02e22c thru 0x032d37 (0x4b0c bytes) 000241 1780336097.002588 read 0x00e806 thru 0x01765f (0x8e5a bytes) 000242 1780336097.039474 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ef86): Mirror 3: 0xbec6083f (stale) Mirror 4: 0xbec6083f (stale) Mirror 5: 0xbec6083f (stale) Mirror 6: 0xbec6083f (stale) Mirror 7: 0xbec6083f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef86): Mirror 3: 0xbec6083f (stale) Mirror 4: 0xbec6083f (stale) Mirror 5: 0xbec6083f (stale) Mirror 6: 0xbec6083f (stale) Mirror 7: 0xbec6083f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef86: skipped 000243 1780336097.446567 fallocate 0x0243f4 thru 0x028418 (0x4025 bytes) 000244 1780336097.454818 fallocate 0x035f1b thru 0x03ffff (0xa0e5 bytes) 000245 1780336097.466292 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000246 1780336097.941950 write 0x012a76 thru 0x01c54b (0x9ad6 bytes) 000247 1780336097.983774 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000248 1780336098.544880 fallocate 0x0120db thru 0x01d770 (0xb696 bytes) 000249 1780336098.553949 write 0x014f9d thru 0x024a6d (0xfad1 bytes) 000250 1780336098.574179 write 0x01f112 thru 0x01f65a (0x549 bytes) 000253 1780336098.575999 mapwrite 0x02c210 thru 0x0378e6 (0xb6d7 bytes) 000254 1780336098.630252 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000255 1780336098.948406 fallocate 0x0157f6 thru 0x01e1f2 (0x89fd bytes) 000256 1780336098.954944 fallocate 0x019c27 thru 0x01af5d (0x1337 bytes) 000257 1780336098.960278 fallocate 0x003003 thru 0x012581 (0xf57f bytes) 000258 1780336098.966265 write 0x011e64 thru 0x01916c (0x7309 bytes) 000259 1780336098.983776 punch from 0x019a83 to 0x02361c (0x9b9a bytes) 000260 1780336099.004938 punch from 0x009e4c to 0x00d397 (0x354c bytes) 000262 1780336099.017974 write 0x0270b4 thru 0x03029c (0x91e9 bytes) 000263 1780336099.074526 fallocate 0x0256c0 thru 0x0274ab (0x1dec bytes) 000265 1780336099.081498 trunc from 0x02ab86 to 0x034c33 (0xa0ae bytes) 000266 1780336099.095702 trunc from 0x012e0a to 0x0140cd (0x12c4 bytes) 000267 1780336099.101114 mapwrite 0x023b5d thru 0x027353 (0x37f7 bytes) 000268 1780336099.120087 read 0x01e342 thru 0x0295de (0xb29d bytes) 000269 1780336099.128645 read 0x01934c thru 0x029277 (0xff2c bytes) 000270 1780336099.150535 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xecd5b7c8 Mirror 8: 0xecd5b7c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000271 1780336099.966536 read 0x023764 thru 0x03259c (0xee39 bytes) 000272 1780336099.978377 trunc from 0x00a7f8 to 0x0108fd (0x6106 bytes) 000273 1780336100.046006 mapwrite 0x03af00 thru 0x03ffff (0x5100 bytes) 000274 1780336100.106521 punch from 0x004156 to 0x0047a9 (0x654 bytes) 000275 1780336100.110872 trunc from 0x0013dd to 0x00f087 (0xdcab bytes) 000276 1780336100.117485 punch from 0x035a9a to 0x03ba9c (0x6003 bytes) 000278 1780336100.125825 punch from 0x02dcb1 to 0x03ccd9 (0xf029 bytes) 000279 1780336100.132166 trunc from 0x03ea22 to 0x03fde7 (0x13c6 bytes) 000280 1780336100.139441 read 0x01eed5 thru 0x027382 (0x84ae bytes) 000281 1780336100.160292 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 8: 0xecd5b7c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 8: 0xecd5b7c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000282 1780336100.674927 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 8: 0xecd5b7c8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc0f9c7e3 Mirror 9: 0xc0f9c7e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000284 1780336101.676811 trunc from 0x040000 to 0x00fa63 (0x3059c bytes) 000285 1780336101.729778 write 0x036621 thru 0x03af81 (0x4961 bytes) 000286 1780336101.732483 read 0x00fc16 thru 0x01cf67 (0xd352 bytes) 000288 1780336101.748549 mapwrite 0x03adac thru 0x03ffff (0x5254 bytes) 000289 1780336101.786404 mapread 0x02dab0 thru 0x03d892 (0xfde3 bytes) 000290 1780336101.792549 mapread 0x03a0c8 thru 0x03d260 (0x3199 bytes) 000291 1780336101.793400 trunc from 0x0212eb to 0x021972 (0x688 bytes) 000292 1780336101.802926 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 8: 0xecd5b7c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 8: 0xecd5b7c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000293 1780336102.132339 read 0x02fe26 thru 0x03ee4f (0xf02a bytes) 000294 1780336102.145631 fallocate 0x00db29 thru 0x0197f1 (0xbcc9 bytes) 000295 1780336102.151049 mapwrite 0x002e0f thru 0x00ae39 (0x802b bytes) 000296 1780336102.200531 trunc from 0x040000 to 0x03454b (0xbab4 bytes) 000297 1780336102.227223 read 0x00fb52 thru 0x014893 (0x4d42 bytes) 000299 1780336102.240170 trunc from 0x0331df to 0x03454b (0x136d bytes) 000300 1780336102.248202 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3454c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3454c): Mirror 6: 0x60238ab5 (stale) Mirror 8: 0x241ad3b8 (stale) CRC-32 checksum value for chunk [0, 0x3454c): Mirror 1: 0x38929921 Mirror 9: 0x38929921 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3454c: skipped 000301 1780336103.301450 mapread 0x0078a9 thru 0x0136d9 (0xbe31 bytes) 000302 1780336103.333777 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3454c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3454c): Mirror 6: 0x60238ab5 (stale) Mirror 8: 0x241ad3b8 (stale) CRC-32 checksum value for chunk [0, 0x3454c): Mirror 1: 0x38929921 Mirror 9: 0x38929921 Mirror 10: 0x38929921 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3454c: skipped 000303 1780336104.401724 mapwrite 0x00cd8e thru 0x01caa1 (0xfd14 bytes) 000304 1780336104.527277 trunc from 0x015165 to 0x024d4d (0xfbe9 bytes) 000305 1780336104.543380 trunc from 0x026469 to 0x028fa3 (0x2b3b bytes) 000307 1780336104.562822 trunc from 0x03454c to 0x016b02 (0x1da49 bytes) 000308 1780336104.637493 mapread 0x00de5f thru 0x011916 (0x3ab8 bytes) 000309 1780336104.644614 trunc from 0x00e656 to 0x016b02 (0x84ad bytes) 000310 1780336104.658383 trunc from 0x016b03 to 0x01f5df (0x8add bytes) 000311 1780336104.683725 read 0x00374a thru 0x00e88f (0xb146 bytes) 000312 1780336104.694457 mapread 0x01c64c thru 0x01f5df (0x2f94 bytes) 000314 1780336104.708216 trunc from 0x0090db to 0x01697e (0xd8a4 bytes) 000315 1780336104.717458 mapwrite 0x02115e thru 0x023793 (0x2636 bytes) 000316 1780336104.769182 fallocate 0x02b731 thru 0x0315c8 (0x5e98 bytes) 000318 1780336104.776259 write 0x026561 thru 0x02b70a (0x51aa bytes) 000319 1780336104.777637 fallocate 0x01889a thru 0x01f76a (0x6ed1 bytes) 000320 1780336104.785540 punch from 0x02217d to 0x02b70a (0x958e bytes) 000321 1780336104.807644 fallocate 0x0358a0 thru 0x03795e (0x20bf bytes) 000322 1780336104.816429 punch from 0x008526 to 0x00ad1d (0x27f8 bytes) 000323 1780336104.825704 trunc from 0x03795f to 0x002b2f (0x34e2f bytes) 000325 1780336104.850151 punch from 0x0028d8 to 0x002b2f (0x258 bytes) 000326 1780336104.857309 mapwrite 0x00c6ba thru 0x018b6a (0xc4b1 bytes) 000327 1780336104.926251 mapwrite 0x038a1a thru 0x03ffff (0x75e6 bytes) 000328 1780336104.974347 write 0x01b2a7 thru 0x01dac8 (0x2822 bytes) 000329 1780336104.996305 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 8: 0xecd5b7c8 (stale) Mirror 10: 0xcb2f6a17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 8: 0xecd5b7c8 (stale) Mirror 10: 0xcb2f6a17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000330 1780336105.445741 trunc from 0x030052 to 0x0364fe (0x64ad bytes) 000331 1780336105.462369 read 0x00b003 thru 0x0131b5 (0x81b3 bytes) 000334 1780336105.483756 fallocate 0x00f3e5 thru 0x01eeef (0xfb0b bytes) 000335 1780336105.492680 mapwrite 0x038c9c thru 0x03ffff (0x7364 bytes) 000337 1780336105.574115 punch from 0x035a2c to 0x03ffff (0xa5d4 bytes) 000338 1780336105.588236 trunc from 0x040000 to 0x0131d1 (0x2ce2e bytes) 000339 1780336105.626570 fallocate 0x03230c thru 0x037e14 (0x5b09 bytes) 000340 1780336105.636751 trunc from 0x021a7a to 0x02b2ea (0x9871 bytes) 000341 1780336105.643428 trunc from 0x037e15 to 0x010a29 (0x273eb bytes) 000342 1780336105.671457 punch from 0x0108d0 to 0x010a29 (0x15a bytes) 000343 1780336105.695730 punch from 0x009706 to 0x00d2de (0x3bd9 bytes) 000345 1780336105.715797 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x10a2a): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10a2a): Mirror 6: 0x592a0429 (stale) Mirror 8: 0x592a0429 (stale) Mirror 10: 0xb690d14 (stale) CRC-32 checksum value for chunk [0, 0x10a2a): Mirror 1: 0xb4924a8c Mirror 11: 0xb4924a8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10a2a: skipped 000346 1780336107.062100 mapwrite 0x01c086 thru 0x01e5c0 (0x253b bytes) 000347 1780336107.208751 write 0x00589f thru 0x014f53 (0xf6b5 bytes) 000348 1780336107.223445 mapwrite 0x03656e thru 0x0385ea (0x207d bytes) 000349 1780336107.279389 mapwrite 0x01418d thru 0x014cd6 (0xb4a bytes) 000350 1780336107.293558 write 0x03900a thru 0x03cb2a (0x3b21 bytes) 000351 1780336107.294680 fallocate 0x031527 thru 0x034c8a (0x3764 bytes) 000352 1780336107.302331 fallocate 0x01c946 thru 0x02c8e9 (0xffa4 bytes) 000353 1780336107.308688 write 0x011867 thru 0x01bcb5 (0xa44f bytes) 000354 1780336107.311451 mapread 0x01b61a thru 0x0204f0 (0x4ed7 bytes) 000355 1780336107.323141 punch from 0x00d110 to 0x018f67 (0xbe58 bytes) 000357 1780336107.345465 trunc from 0x01be19 to 0x025b3c (0x9d24 bytes) 000358 1780336107.353743 write 0x00164c thru 0x0041a9 (0x2b5e bytes) 000359 1780336107.359587 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cb2b): Mirror 6: 0xc094468d (stale) Mirror 8: 0xbb6b5f8f (stale) Mirror 11: 0x49d9f732 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb2b): Mirror 6: 0xc094468d (stale) Mirror 8: 0xbb6b5f8f (stale) Mirror 11: 0x49d9f732 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb2b: skipped 000361 1780336107.840738 mapread 0x01aae2 thru 0x01d776 (0x2c95 bytes) 000362 1780336107.866112 read 0x039c29 thru 0x03cb2a (0x2f02 bytes) 000363 1780336107.879183 read 0x0306b5 thru 0x03131f (0xc6b bytes) 000364 1780336107.892198 read 0x013715 thru 0x018b2d (0x5419 bytes) 000365 1780336107.893367 fallocate 0x015fba thru 0x021ca3 (0xbcea bytes) 000366 1780336107.904424 punch from 0x0109c4 to 0x0166bc (0x5cf9 bytes) 000368 1780336107.914545 punch from 0x009ad3 to 0x012edb (0x9409 bytes) 000369 1780336107.921724 fallocate 0x0010af thru 0x006aba (0x5a0c bytes) 000370 1780336107.929227 mapwrite 0x03dc0d thru 0x03ffff (0x23f3 bytes) 000371 1780336108.004174 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 8: 0xecd5b7c8 (stale) Mirror 11: 0xd73cf936 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x67694593 Mirror 12: 0x67694593 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000372 1780336108.958256 mapread 0x027dd7 thru 0x034672 (0xc89c bytes) 000373 1780336109.010406 trunc from 0x00e17a to 0x0113b9 (0x3240 bytes) 000374 1780336109.070353 read 0x01ee56 thru 0x01fb8a (0xd35 bytes) 000375 1780336109.077193 mapwrite 0x023b72 thru 0x025788 (0x1c17 bytes) 000376 1780336109.105533 MIRROR_RESYNC 5 mirrors 000377 1780336111.962274 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000378 1780336112.523260 punch from 0x002316 to 0x008d4a (0x6a35 bytes) 000379 1780336112.608928 punch from 0x002442 to 0x003ff0 (0x1baf bytes) 000380 1780336112.649713 trunc from 0x040000 to 0x0103bf (0x2fc40 bytes) 000381 1780336112.707748 mapwrite 0x035c65 thru 0x03c53a (0x68d6 bytes) 000382 1780336112.816802 punch from 0x028cdb to 0x02b398 (0x26be bytes) 000383 1780336112.832615 punch from 0x00abf2 to 0x00bf55 (0x1364 bytes) 000385 1780336112.849159 mapwrite 0x034a20 thru 0x03ffff (0xb5e0 bytes) 000386 1780336112.922812 mapread 0x03846b thru 0x03ffff (0x7b95 bytes) 000387 1780336112.924073 fallocate 0x02f14f thru 0x030a17 (0x18c9 bytes) 000388 1780336112.935511 fallocate 0x007404 thru 0x015545 (0xe142 bytes) 000389 1780336112.941143 mapwrite 0x032cc3 thru 0x038013 (0x5351 bytes) 000390 1780336112.958181 fallocate 0x00b9a6 thru 0x01b1e6 (0xf841 bytes) 000391 1780336112.968120 fallocate 0x004ff7 thru 0x011503 (0xc50d bytes) 000392 1780336112.974517 trunc from 0x040000 to 0x01b205 (0x24dfa bytes) 000393 1780336113.017219 read 0x00eeaa thru 0x014320 (0x5477 bytes) 000394 1780336113.064986 mapread 0x014c23 thru 0x01b205 (0x65e3 bytes) 000395 1780336113.082457 write 0x00c698 thru 0x00f949 (0x32b2 bytes) 000396 1780336113.100100 punch from 0x018126 to 0x01b205 (0x30e0 bytes) 000397 1780336113.113462 MIRROR_RESYNC 4 mirrors 000399 1780336114.507813 trunc from 0x01a7ba to 0x01b205 (0xa4c bytes) 000400 1780336114.548454 mapwrite 0x00a89e thru 0x00f844 (0x4fa7 bytes) 000401 1780336114.592512 punch from 0x0013b9 to 0x00261e (0x1266 bytes) 000402 1780336114.597254 mapwrite 0x010eef thru 0x01d89b (0xc9ad bytes) 000403 1780336114.655302 mapwrite 0x028c30 thru 0x035622 (0xc9f3 bytes) 000404 1780336114.708207 punch from 0x034171 to 0x035622 (0x14b2 bytes) 000405 1780336114.714287 read 0x00580c thru 0x0076c6 (0x1ebb bytes) 000407 1780336114.716510 mapread 0x014489 thru 0x019c8c (0x5804 bytes) 000408 1780336114.718001 mapwrite 0x03c90e thru 0x03ffff (0x36f2 bytes) 000409 1780336114.758259 trunc from 0x040000 to 0x0234a1 (0x1cb5e bytes) 000410 1780336114.780792 read 0x01ca11 thru 0x0234a1 (0x6a91 bytes) 000411 1780336114.790073 trunc from 0x0207bd to 0x0234a1 (0x2ce5 bytes) 000412 1780336114.795054 read 0x00e066 thru 0x00fcbf (0x1c5a bytes) 000413 1780336114.796223 punch from 0x01456a to 0x022967 (0xe3fe bytes) 000414 1780336114.801673 mapwrite 0x016fb8 thru 0x020341 (0x938a bytes) 000415 1780336114.835325 mapwrite 0x02025c thru 0x026fb2 (0x6d57 bytes) 000416 1780336114.879205 fallocate 0x024a2c thru 0x02c8e4 (0x7eb9 bytes) 000418 1780336114.888052 read 0x0087b9 thru 0x0096fc (0xf44 bytes) 000419 1780336114.889218 mapwrite 0x0223bd thru 0x0291d2 (0x6e16 bytes) 000420 1780336114.918748 trunc from 0x01ada2 to 0x01b753 (0x9b2 bytes) 000421 1780336114.925182 punch from 0x006011 to 0x006a83 (0xa73 bytes) 000422 1780336114.930440 punch from 0x00d189 to 0x011b23 (0x499b bytes) 000423 1780336114.937355 fallocate 0x027008 thru 0x02f471 (0x846a bytes) 000424 1780336114.944105 read 0x022e8c thru 0x02c8e4 (0x9a59 bytes) 000425 1780336114.945353 mapwrite 0x0075ad thru 0x013eb1 (0xc905 bytes) 000426 1780336114.987163 trunc from 0x02c8e5 to 0x01e630 (0xe2b4 bytes) 000427 1780336115.008259 trunc from 0x01e631 to 0x00c98d (0x11ca3 bytes) 000428 1780336115.034720 read 0x00405b thru 0x007ab9 (0x3a5f bytes) 000429 1780336115.036223 mapwrite 0x02e671 thru 0x03884b (0xa1db bytes) 000430 1780336115.128238 fallocate 0x00f31c thru 0x015a6c (0x6751 bytes) 000431 1780336115.137223 trunc from 0x03884c to 0x01d322 (0x1b529 bytes) 000432 1780336115.159580 write 0x00299c thru 0x00ea83 (0xc0e8 bytes) 000433 1780336115.161703 mapwrite 0x00e924 thru 0x01287b (0x3f58 bytes) 000435 1780336115.187130 write 0x01cf62 thru 0x0240ef (0x718e bytes) 000436 1780336115.195098 MIRROR_RESYNC 4 mirrors 000437 1780336116.130000 mapwrite 0x00f965 thru 0x016c2f (0x72cb bytes) 000438 1780336116.217827 punch from 0x0105dd to 0x01fa23 (0xf447 bytes) 000439 1780336116.224216 punch from 0x01c266 to 0x0240ef (0x7e8a bytes) 000440 1780336116.231169 trunc from 0x007757 to 0x00eae0 (0x738a bytes) 000441 1780336116.236737 trunc from 0x0240f0 to 0x00cc44 (0x174ab bytes) 000442 1780336116.284288 trunc from 0x00490c to 0x00cc09 (0x82fe bytes) 000443 1780336116.291025 mapwrite 0x03f697 thru 0x03ffff (0x969 bytes) 000444 1780336116.331774 read 0x0273c2 thru 0x0290e7 (0x1d26 bytes) 000445 1780336116.337725 mapread 0x02c35f thru 0x036399 (0xa03b bytes) 000446 1780336116.362522 mapwrite 0x003d45 thru 0x00e080 (0xa33c bytes) 000447 1780336116.399030 read 0x016747 thru 0x01cc00 (0x64ba bytes) 000449 1780336116.411138 punch from 0x01cf9e to 0x01ecfe (0x1d61 bytes) 000450 1780336116.416177 fallocate 0x020791 thru 0x02749a (0x6d0a bytes) 000451 1780336116.422497 trunc from 0x040000 to 0x009ec0 (0x3613f bytes) 000452 1780336116.446137 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x9ec1): Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9ec1): Mirror 6: 0x25e7e2be (stale) Mirror 11: 0x25e7e2be (stale) Mirror 12: 0x25e7e2be (stale) CRC-32 checksum value for chunk [0, 0x9ec1): Mirror 1: 0x49da21e9 Mirror 13: 0x49da21e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9ec1: skipped 000455 1780336117.387795 fallocate 0x0264d0 thru 0x02db8f (0x76c0 bytes) 000456 1780336117.453080 trunc from 0x009ec1 to 0x03560f (0x2b74f bytes) 000457 1780336117.513572 fallocate 0x0123b3 thru 0x01ac91 (0x88df bytes) 000458 1780336117.525769 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35610): Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35610): Mirror 6: 0xfaafe9c3 (stale) Mirror 11: 0xfaafe9c3 (stale) Mirror 12: 0xfaafe9c3 (stale) Mirror 13: 0x16710f3b (stale) CRC-32 checksum value for chunk [0, 0x35610): Mirror 1: 0x16710f3b Mirror 14: 0x16710f3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35610: skipped 000459 1780336118.658390 punch from 0x00f2f2 to 0x01222e (0x2f3d bytes) 000460 1780336118.708383 write 0x022ccd thru 0x02f52e (0xc862 bytes) 000461 1780336118.725594 mapread 0x01a13b thru 0x023c00 (0x9ac6 bytes) 000462 1780336118.736670 fallocate 0x031a8c thru 0x037a91 (0x6006 bytes) 000463 1780336118.748164 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x37a92): Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a92): Mirror 6: 0x25fe4691 (stale) Mirror 11: 0x25fe4691 (stale) Mirror 12: 0x25fe4691 (stale) Mirror 13: 0xfb8da2e8 (stale) Mirror 14: 0xfb8da2e8 (stale) CRC-32 checksum value for chunk [0, 0x37a92): Mirror 1: 0xaf5aff92 Mirror 15: 0xaf5aff92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a92: skipped 000464 1780336120.231932 MIRROR_RESYNC 7 mirrors 000465 1780336122.407976 read 0x0351a1 thru 0x037a91 (0x28f1 bytes) 000466 1780336122.413562 fallocate 0x03edd2 thru 0x03ffff (0x122e bytes) 000467 1780336122.476674 write 0x00a8ad thru 0x012078 (0x77cc bytes) 000468 1780336122.511114 read 0x02f781 thru 0x034156 (0x49d6 bytes) 000469 1780336122.539602 read 0x03f43c thru 0x03ffff (0xbc4 bytes) 000470 1780336122.540306 trunc from 0x040000 to 0x02cff1 (0x1300e bytes) 000471 1780336122.576429 read 0x0082c4 thru 0x008ba1 (0x8de bytes) 000472 1780336122.578522 punch from 0x012af3 to 0x01e9de (0xbeec bytes) 000473 1780336122.598678 mapread 0x00ba3a thru 0x0103c9 (0x4990 bytes) 000474 1780336122.600642 mapwrite 0x0168ce thru 0x0259a6 (0xf0d9 bytes) 000475 1780336122.652407 mapread 0x029440 thru 0x02cff1 (0x3bb2 bytes) 000476 1780336122.655112 mapread 0x001a74 thru 0x010ad6 (0xf063 bytes) 000477 1780336122.658936 punch from 0x016f2c to 0x01a553 (0x3628 bytes) 000478 1780336122.666519 fallocate 0x0317d4 thru 0x03621f (0x4a4c bytes) 000479 1780336122.673045 punch from 0x012805 to 0x012efe (0x6fa bytes) 000480 1780336122.679559 read 0x034c3b thru 0x03621f (0x15e5 bytes) 000481 1780336122.691078 fallocate 0x01a7df thru 0x0284ee (0xdd10 bytes) 000482 1780336122.697552 mapwrite 0x02daf7 thru 0x038e9f (0xb3a9 bytes) 000483 1780336122.746818 write 0x0256aa thru 0x02ff67 (0xa8be bytes) 000484 1780336122.752178 mapwrite 0x01339f thru 0x0166ea (0x334c bytes) 000485 1780336122.777516 punch from 0x02015c to 0x020963 (0x808 bytes) 000486 1780336122.787010 write 0x003781 thru 0x0100b1 (0xc931 bytes) 000487 1780336122.789829 trunc from 0x038ea0 to 0x0252eb (0x13bb4 bytes) 000488 1780336122.825089 mapread 0x005402 thru 0x00cfe3 (0x7be2 bytes) 000491 1780336122.836993 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x252ec): Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x252ec): Mirror 6: 0x3087f54c (stale) Mirror 11: 0x3087f54c (stale) Mirror 12: 0x3087f54c (stale) Mirror 13: 0x3087f54c (stale) Mirror 14: 0x3087f54c (stale) Mirror 15: 0x3087f54c (stale) CRC-32 checksum value for chunk [0, 0x252ec): Mirror 1: 0xf183e82d Mirror 16: 0xf183e82d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x252ec: skipped 000492 1780336124.012143 trunc from 0x0252ec to 0x029e1f (0x4b34 bytes) 000494 1780336124.084964 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29e20): Mirror 6: 0x7bd2cb40 (stale) Mirror 11: 0x7bd2cb40 (stale) Mirror 12: 0x7bd2cb40 (stale) Mirror 13: 0x7bd2cb40 (stale) Mirror 15: 0x7bd2cb40 (stale) Mirror 16: 0xa361ab15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29e20): Mirror 6: 0x7bd2cb40 (stale) Mirror 11: 0x7bd2cb40 (stale) Mirror 12: 0x7bd2cb40 (stale) Mirror 13: 0x7bd2cb40 (stale) Mirror 15: 0x7bd2cb40 (stale) Mirror 16: 0xa361ab15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29e20: skipped 000495 1780336124.718133 write 0x010a2c thru 0x01c69a (0xbc6f bytes) 000496 1780336124.752818 mapread 0x01a7f2 thru 0x01fe32 (0x5641 bytes) 000497 1780336124.775376 fallocate 0x00187d thru 0x00d593 (0xbd17 bytes) 000498 1780336124.782083 fallocate 0x030e05 thru 0x03d731 (0xc92d bytes) 000499 1780336124.826099 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x29e20): Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29e20): Mirror 6: 0x7bd2cb40 (stale) Mirror 11: 0x7bd2cb40 (stale) Mirror 12: 0x7bd2cb40 (stale) Mirror 13: 0x7bd2cb40 (stale) Mirror 15: 0x7bd2cb40 (stale) Mirror 16: 0xa361ab15 (stale) CRC-32 checksum value for chunk [0, 0x29e20): Mirror 1: 0x66a88fe6 Mirror 17: 0x66a88fe6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29e20: skipped 000500 1780336126.302529 mapread 0x0202cf thru 0x029e1f (0x9b51 bytes) 000501 1780336126.326591 mapread 0x01fea3 thru 0x029e1f (0x9f7d bytes) 000502 1780336126.328053 fallocate 0x01ad9a thru 0x0229f5 (0x7c5c bytes) 000503 1780336126.384705 read 0x00b9ae thru 0x012a43 (0x7096 bytes) 000504 1780336126.397099 read 0x010b57 thru 0x01ad63 (0xa20d bytes) 000505 1780336126.416071 trunc from 0x029e20 to 0x014a95 (0x1538a bytes) 000506 1780336126.451817 mapread 0x007060 thru 0x008278 (0x1219 bytes) 000507 1780336126.453013 trunc from 0x014a96 to 0x02346d (0xe9d8 bytes) 000508 1780336126.471463 write 0x0198b6 thru 0x0285d2 (0xed1d bytes) 000509 1780336126.484620 mapread 0x0249ab thru 0x0285d2 (0x3c28 bytes) 000510 1780336126.486322 read 0x006783 thru 0x00c07e (0x58fc bytes) 000511 1780336126.494361 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x285d3): Mirror 6: 0x8fb4ac9f (stale) Mirror 11: 0x8fb4ac9f (stale) Mirror 12: 0x8fb4ac9f (stale) Mirror 13: 0x8fb4ac9f (stale) Mirror 15: 0x8fb4ac9f (stale) Mirror 17: 0x92b39f8a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x285d3): Mirror 6: 0x8fb4ac9f (stale) Mirror 11: 0x8fb4ac9f (stale) Mirror 12: 0x8fb4ac9f (stale) Mirror 13: 0x8fb4ac9f (stale) Mirror 15: 0x8fb4ac9f (stale) Mirror 17: 0x92b39f8a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x285d3: skipped 000512 1780336127.103488 write 0x00d519 thru 0x0179ec (0xa4d4 bytes) 000513 1780336127.139573 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x285d3): Mirror 6: 0x8fb4ac9f (stale) Mirror 12: 0x8fb4ac9f (stale) Mirror 13: 0x8fb4ac9f (stale) Mirror 15: 0x8fb4ac9f (stale) Mirror 17: 0x92b39f8a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x285d3): Mirror 6: 0x8fb4ac9f (stale) Mirror 12: 0x8fb4ac9f (stale) Mirror 13: 0x8fb4ac9f (stale) Mirror 15: 0x8fb4ac9f (stale) Mirror 17: 0x92b39f8a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x285d3: skipped 000514 1780336127.660391 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x285d3): Mirror 6: 0x8fb4ac9f (stale) Mirror 12: 0x8fb4ac9f (stale) Mirror 13: 0x8fb4ac9f (stale) Mirror 17: 0x92b39f8a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x285d3): Mirror 6: 0x8fb4ac9f (stale) Mirror 12: 0x8fb4ac9f (stale) Mirror 13: 0x8fb4ac9f (stale) Mirror 17: 0x92b39f8a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x285d3: skipped 000515 1780336128.246520 read 0x00feca thru 0x01787a (0x79b1 bytes) 000516 1780336128.268725 trunc from 0x0285d3 to 0x020591 (0x8041 bytes) 000517 1780336128.298564 mapwrite 0x02d3f1 thru 0x031c6c (0x487c bytes) 000518 1780336128.353589 trunc from 0x031c6d to 0x00b345 (0x26927 bytes) 000519 1780336128.380702 write 0x024276 thru 0x02df1b (0x9ca6 bytes) 000520 1780336128.391588 MIRROR_RESYNC 5 mirrors 000521 1780336129.947544 fallocate 0x030193 thru 0x03d23f (0xd0ad bytes) 000522 1780336129.985264 trunc from 0x012fbd to 0x01a98f (0x79d3 bytes) 000523 1780336129.991268 trunc from 0x03d240 to 0x02059a (0x1cca5 bytes) 000524 1780336130.015168 mapwrite 0x02a4eb thru 0x034e47 (0xa95d bytes) 000525 1780336130.086446 punch from 0x002bb0 to 0x01153f (0xe990 bytes) 000526 1780336130.093548 mapread 0x023bce thru 0x030842 (0xcc75 bytes) 000527 1780336130.104275 fallocate 0x0391e3 thru 0x0393f5 (0x213 bytes) 000528 1780336130.110466 read 0x00b164 thru 0x00c7c6 (0x1663 bytes) 000529 1780336130.116686 mapread 0x00a44b thru 0x01144c (0x7002 bytes) 000530 1780336130.127991 trunc from 0x034e48 to 0x00eacf (0x26378 bytes) 000531 1780336130.147888 write 0x0007c2 thru 0x000920 (0x15f bytes) 000532 1780336130.155063 trunc from 0x0058cb to 0x00c7dd (0x6f13 bytes) 000533 1780336130.160753 fallocate 0x035eb4 thru 0x03ffff (0xa14c bytes) 000534 1780336130.166578 write 0x02f0a9 thru 0x0396ff (0xa657 bytes) 000535 1780336130.170340 mapread 0x02cc7f thru 0x03388c (0x6c0e bytes) 000536 1780336130.180186 write 0x02ffcc thru 0x037cad (0x7ce2 bytes) 000537 1780336130.182699 fallocate 0x037048 thru 0x03def8 (0x6eb1 bytes) 000538 1780336130.188311 mapread 0x02494f thru 0x024fe9 (0x69b bytes) 000540 1780336130.197037 fallocate 0x000b1a thru 0x00b291 (0xa778 bytes) 000541 1780336130.203587 fallocate 0x03eefc thru 0x03ffff (0x1104 bytes) 000543 1780336130.210130 read 0x02c346 thru 0x02fdd4 (0x3a8f bytes) 000544 1780336130.212175 mapwrite 0x02b530 thru 0x033445 (0x7f16 bytes) 000545 1780336130.233887 trunc from 0x039700 to 0x02ab8b (0xeb74 bytes) 000546 1780336130.254513 punch from 0x021553 to 0x02ab8b (0x9639 bytes) 000548 1780336130.266676 MIRROR_RESYNC 5 mirrors 000549 1780336131.168608 fallocate 0x031abd thru 0x03827d (0x67c1 bytes) 000550 1780336131.209935 read 0x028a5d thru 0x0316dd (0x8c81 bytes) 000552 1780336131.220562 trunc from 0x03827e to 0x00e759 (0x29b24 bytes) 000553 1780336131.250336 fallocate 0x00ce3e thru 0x01c740 (0xf903 bytes) 000554 1780336131.262064 MIRROR_RESYNC 5 mirrors 000555 1780336132.428385 mapread 0x0119e5 thru 0x01c740 (0xad5c bytes) 000557 1780336132.460611 write 0x01c1f1 thru 0x01df00 (0x1d10 bytes) 000558 1780336132.541893 trunc from 0x01335d to 0x013e89 (0xb2d bytes) 000559 1780336132.548474 punch from 0x002bb4 to 0x008b67 (0x5fb4 bytes) 000560 1780336132.556387 trunc from 0x01df01 to 0x01b70d (0x27f3 bytes) 000561 1780336132.633014 trunc from 0x00c2ba to 0x01ae74 (0xebbb bytes) 000562 1780336132.638679 trunc from 0x00de0a to 0x015afc (0x7cf3 bytes) 000563 1780336132.649909 MIRROR_RESYNC 5 mirrors 000564 1780336133.602705 MIRROR_RESYNC 5 mirrors 000566 1780336134.079325 mapread 0x019e38 thru 0x01b70d (0x18d6 bytes) 000567 1780336134.102289 punch from 0x005bda to 0x00c4b1 (0x68d8 bytes) 000568 1780336134.144208 trunc from 0x00c81f to 0x00d871 (0x1053 bytes) 000569 1780336134.150387 mapread 0x00d5e8 thru 0x01758d (0x9fa6 bytes) 000570 1780336134.179638 trunc from 0x00487f to 0x00d31f (0x8aa1 bytes) 000571 1780336134.187302 write 0x00c5f4 thru 0x017237 (0xac44 bytes) 000572 1780336134.200290 mapwrite 0x00eafb thru 0x01dae3 (0xefe9 bytes) 000574 1780336134.285035 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dae4): Mirror 12: 0x52e61496 (stale) Mirror 13: 0x52e61496 (stale) Mirror 17: 0x52e61496 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dae4): Mirror 12: 0x52e61496 (stale) Mirror 13: 0x52e61496 (stale) Mirror 17: 0x52e61496 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dae4: skipped 000575 1780336134.787166 trunc from 0x00f5fb to 0x01dae3 (0xe4e9 bytes) 000576 1780336134.795154 write 0x01e405 thru 0x0235aa (0x51a6 bytes) 000577 1780336134.801162 trunc from 0x0235ab to 0x039c2b (0x16681 bytes) 000578 1780336134.817355 trunc from 0x01c6c2 to 0x01cbbe (0x4fd bytes) 000579 1780336134.826821 mapwrite 0x03724a thru 0x038af7 (0x18ae bytes) 000580 1780336134.884346 trunc from 0x01dcd6 to 0x02508c (0x73b7 bytes) 000581 1780336134.889791 read 0x007e6b thru 0x01244f (0xa5e5 bytes) 000582 1780336134.901216 read 0x0269ed thru 0x034e9f (0xe4b3 bytes) 000584 1780336134.915807 mapread 0x00bd02 thru 0x0172a7 (0xb5a6 bytes) 000585 1780336134.918739 mapread 0x02ccc9 thru 0x039c2b (0xcf63 bytes) 000586 1780336134.921386 trunc from 0x039c2c to 0x00ffd4 (0x29c57 bytes) 000587 1780336134.948357 MIRROR_RESYNC 4 mirrors 000588 1780336135.853928 MIRROR_EXTEND to 5 mirrors 000589 1780336136.912359 write 0x00cc01 thru 0x015781 (0x8b81 bytes) 000590 1780336136.974899 trunc from 0x01276d to 0x01437d (0x1c11 bytes) 000591 1780336137.023228 trunc from 0x00e7d0 to 0x015781 (0x6fb2 bytes) 000592 1780336137.038358 mapwrite 0x00544c thru 0x00cc8b (0x7840 bytes) 000593 1780336137.129792 write 0x033791 thru 0x034056 (0x8c6 bytes) 000595 1780336137.143397 trunc from 0x030d52 to 0x034056 (0x3305 bytes) 000596 1780336137.160197 punch from 0x02d48e to 0x034056 (0x6bc9 bytes) 000597 1780336137.176541 MIRROR_RESYNC 5 mirrors 000598 1780336138.796693 punch from 0x0047d7 to 0x01055d (0xbd87 bytes) 000599 1780336138.831270 trunc from 0x034057 to 0x02460f (0xfa47 bytes) 000600 1780336138.854703 fallocate 0x02f7b7 thru 0x03daae (0xe2f8 bytes) 000601 1780336138.860743 mapwrite 0x020141 thru 0x025f55 (0x5e15 bytes) 000602 1780336138.897390 fallocate 0x00627b thru 0x01218b (0xbf11 bytes) 000603 1780336138.903585 mapread 0x017ca9 thru 0x024ad5 (0xce2d bytes) 000604 1780336138.912725 trunc from 0x03daaf to 0x036708 (0x73a6 bytes) 000605 1780336138.930449 fallocate 0x03df61 thru 0x03ffff (0x209f bytes) 000606 1780336138.944666 write 0x00c369 thru 0x0179c5 (0xb65d bytes) 000607 1780336138.983978 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x4f2f6e1b (stale) Mirror 13: 0x4f2f6e1b (stale) Mirror 17: 0x4f2f6e1b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x4f2f6e1b (stale) Mirror 13: 0x4f2f6e1b (stale) Mirror 17: 0x4f2f6e1b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000608 1780336139.643485 fallocate 0x03d2fa thru 0x03ffff (0x2d06 bytes) 000609 1780336139.656419 fallocate 0x03c419 thru 0x03ffff (0x3be7 bytes) 000610 1780336139.666950 fallocate 0x02685f thru 0x02e1a5 (0x7947 bytes) 000611 1780336139.692093 punch from 0x02b263 to 0x033a52 (0x87f0 bytes) 000612 1780336139.707887 mapread 0x00283d thru 0x010748 (0xdf0c bytes) 000613 1780336139.770137 trunc from 0x030e46 to 0x03ffff (0xf1ba bytes) 000614 1780336139.780156 fallocate 0x0191e4 thru 0x01f477 (0x6294 bytes) 000615 1780336139.794770 mapwrite 0x012e69 thru 0x0206d7 (0xd86f bytes) 000616 1780336139.898400 mapwrite 0x011049 thru 0x016ad9 (0x5a91 bytes) 000617 1780336139.927380 fallocate 0x030488 thru 0x03d936 (0xd4af bytes) 000618 1780336139.938120 fallocate 0x0306aa thru 0x03e984 (0xe2db bytes) 000619 1780336139.947084 trunc from 0x02028d to 0x0208df (0x653 bytes) 000621 1780336139.955320 trunc from 0x040000 to 0x018015 (0x27fea bytes) 000622 1780336139.986632 punch from 0x014c8f to 0x018015 (0x3387 bytes) 000623 1780336139.996087 mapwrite 0x03f992 thru 0x03ffff (0x66e bytes) 000624 1780336140.057186 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x4f2f6e1b (stale) Mirror 13: 0x4f2f6e1b (stale) Mirror 17: 0x4f2f6e1b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe5b8f593 Mirror 18: 0xe5b8f593 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000625 1780336141.690248 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x4f2f6e1b (stale) Mirror 17: 0x4f2f6e1b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x4f2f6e1b (stale) Mirror 17: 0x4f2f6e1b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe5b8f593 Mirror 18: 0xe5b8f593 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000626 1780336142.529597 read 0x028313 thru 0x02ca24 (0x4712 bytes) 000628 1780336142.549510 trunc from 0x040000 to 0x036201 (0x9dfe bytes) 000629 1780336142.743147 mapwrite 0x01f884 thru 0x01fbf1 (0x36e bytes) 000630 1780336142.769373 fallocate 0x02f28c thru 0x03bf76 (0xcceb bytes) 000631 1780336142.777357 punch from 0x0038fe to 0x00f112 (0xb815 bytes) 000632 1780336142.782991 punch from 0x02a225 to 0x02bd55 (0x1b31 bytes) 000633 1780336142.789049 mapread 0x023b28 thru 0x033a8f (0xff68 bytes) 000634 1780336142.834185 trunc from 0x03bf77 to 0x03fdf2 (0x3e7c bytes) 000635 1780336142.877623 mapread 0x02b38d thru 0x02e698 (0x330c bytes) 000636 1780336142.887634 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fdf3): Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fdf3): Mirror 13: 0xa226ec88 (stale) Mirror 17: 0xa226ec88 (stale) Mirror 18: 0xeda68f6 (stale) CRC-32 checksum value for chunk [0, 0x3fdf3): Mirror 1: 0x17d15cf6 Mirror 19: 0x17d15cf6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fdf3: skipped 000637 1780336144.391414 fallocate 0x025c8c thru 0x02cf2c (0x72a1 bytes) 000638 1780336144.461270 mapread 0x022487 thru 0x02973d (0x72b7 bytes) 000639 1780336144.509156 read 0x032412 thru 0x03fdf2 (0xd9e1 bytes) 000640 1780336144.525483 mapwrite 0x01791b thru 0x0210dc (0x97c2 bytes) 000641 1780336144.584744 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fdf3): Mirror 13: 0xa226ec88 (stale) Mirror 17: 0xa226ec88 (stale) Mirror 18: 0xeda68f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fdf3): Mirror 13: 0xa226ec88 (stale) Mirror 17: 0xa226ec88 (stale) Mirror 18: 0xeda68f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fdf3: skipped 000642 1780336145.245084 trunc from 0x03fdf3 to 0x02c817 (0x135db bytes) 000645 1780336145.277064 fallocate 0x0159ed thru 0x01d296 (0x78aa bytes) 000646 1780336145.285272 trunc from 0x02c818 to 0x03dd19 (0x11502 bytes) 000647 1780336145.306640 mapwrite 0x03ea03 thru 0x03ffff (0x15fd bytes) 000648 1780336145.353636 trunc from 0x040000 to 0x02b2bd (0x14d42 bytes) 000649 1780336145.365074 trunc from 0x02b2be to 0x019d31 (0x1158c bytes) 000650 1780336145.378986 write 0x02e20b thru 0x039848 (0xb63e bytes) 000651 1780336145.380599 trunc from 0x039849 to 0x03d259 (0x3a11 bytes) 000652 1780336145.405203 trunc from 0x02dc9e to 0x03b2cd (0xd630 bytes) 000653 1780336145.459710 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d25a): Mirror 13: 0x5560a8e3 (stale) Mirror 18: 0x433ab539 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d25a): Mirror 13: 0x5560a8e3 (stale) Mirror 18: 0x433ab539 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d25a: skipped 000654 1780336146.034241 trunc from 0x01ba1f to 0x0245f5 (0x8bd7 bytes) 000655 1780336146.044710 mapwrite 0x02969a thru 0x036470 (0xcdd7 bytes) 000656 1780336146.141757 MIRROR_RESYNC 3 mirrors 000657 1780336147.945489 fallocate 0x00b8bb thru 0x013cee (0x8434 bytes) 000658 1780336147.993138 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d25a): Mirror 13: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d25a): Mirror 13: 0xa300ea15 (stale) Mirror 18: 0xa300ea15 (stale) CRC-32 checksum value for chunk [0, 0x3d25a): Mirror 1: 0xa300ea15 Mirror 19: 0xa300ea15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d25a: skipped 000659 1780336149.300273 MIRROR_RESYNC 4 mirrors 000660 1780336150.779521 mapwrite 0x021c17 thru 0x0306ce (0xeab8 bytes) 000661 1780336150.980563 fallocate 0x035a8c thru 0x03aaa9 (0x501e bytes) 000662 1780336150.988700 read 0x02cc50 thru 0x0364b3 (0x9864 bytes) 000663 1780336150.996117 read 0x00ba7a thru 0x0108ac (0x4e33 bytes) 000664 1780336151.008052 trunc from 0x02f80a to 0x03a4cd (0xacc4 bytes) 000665 1780336151.019445 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d25a): Mirror 13: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d25a): Mirror 13: 0xa300ea15 (stale) Mirror 18: 0xa300ea15 (stale) Mirror 19: 0xa300ea15 (stale) CRC-32 checksum value for chunk [0, 0x3d25a): Mirror 1: 0x8f513a2c Mirror 20: 0x8f513a2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d25a: skipped 000666 1780336152.374800 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d25a): Mirror 13: 0xa300ea15 (stale) Mirror 18: 0xa300ea15 (stale) Mirror 19: 0xa300ea15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d25a): Mirror 13: 0xa300ea15 (stale) Mirror 18: 0xa300ea15 (stale) Mirror 19: 0xa300ea15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d25a: skipped 000667 1780336152.822328 trunc from 0x02580e to 0x029108 (0x38fb bytes) 000668 1780336152.868971 trunc from 0x02f3a4 to 0x033e9d (0x4afa bytes) 000669 1780336152.878956 write 0x027ff0 thru 0x032e4f (0xae60 bytes) 000670 1780336152.917603 trunc from 0x00ee08 to 0x013f53 (0x514c bytes) 000671 1780336152.930330 write 0x01e968 thru 0x027234 (0x88cd bytes) 000672 1780336152.956698 fallocate 0x01be51 thru 0x02427e (0x842e bytes) 000673 1780336152.964528 write 0x001894 thru 0x0117e0 (0xff4d bytes) 000674 1780336152.985508 fallocate 0x02e956 thru 0x02f250 (0x8fb bytes) 000675 1780336152.995557 write 0x031edf thru 0x0340b8 (0x21da bytes) 000676 1780336153.011562 mapread 0x01355f thru 0x014d87 (0x1829 bytes) 000677 1780336153.033156 MIRROR_RESYNC 4 mirrors 000678 1780336154.748969 fallocate 0x01adbe thru 0x0284d0 (0xd713 bytes) 000679 1780336154.794517 trunc from 0x017311 to 0x01efd1 (0x7cc1 bytes) 000680 1780336154.800439 mapwrite 0x0236b3 thru 0x029b62 (0x64b0 bytes) 000681 1780336154.838384 mapread 0x03a982 thru 0x03d259 (0x28d8 bytes) 000682 1780336154.856393 punch from 0x02a9d2 to 0x02aed4 (0x503 bytes) 000683 1780336154.872790 punch from 0x024df5 to 0x02b292 (0x649e bytes) 000684 1780336154.886786 mapread 0x016d24 thru 0x01bed6 (0x51b3 bytes) 000685 1780336154.905816 mapread 0x0293fe thru 0x038a20 (0xf623 bytes) 000686 1780336154.929422 trunc from 0x03d25a to 0x019e02 (0x23457 bytes) 000687 1780336154.964444 trunc from 0x019e03 to 0x02b4e4 (0x116e2 bytes) 000688 1780336154.981479 read 0x026680 thru 0x02929d (0x2c1e bytes) 000689 1780336154.994510 MIRROR_RESYNC 4 mirrors 000690 1780336156.055510 MIRROR_RESYNC 4 mirrors 000691 1780336156.514724 mapwrite 0x00a35a thru 0x00e6db (0x4382 bytes) 000693 1780336156.717089 mapwrite 0x004eee thru 0x00b862 (0x6975 bytes) 000694 1780336156.741365 mapwrite 0x01c98d thru 0x02ad5c (0xe3d0 bytes) 000695 1780336156.784315 fallocate 0x01da51 thru 0x025e38 (0x83e8 bytes) 000697 1780336156.791055 trunc from 0x0077ea to 0x007b8c (0x3a3 bytes) 000699 1780336156.797804 punch from 0x01df92 to 0x01e23a (0x2a9 bytes) 000700 1780336156.803384 trunc from 0x02b4e5 to 0x0272bb (0x4229 bytes) 000701 1780336156.831103 trunc from 0x0272bc to 0x01643b (0x10e80 bytes) 000702 1780336156.850254 fallocate 0x03f9ac thru 0x03ffff (0x654 bytes) 000703 1780336156.855920 trunc from 0x031cc9 to 0x036a58 (0x4d90 bytes) 000704 1780336156.861458 trunc from 0x00fb79 to 0x0194f2 (0x997a bytes) 000705 1780336156.876740 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x7298c32e (stale) Mirror 19: 0x7298c32e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x7298c32e (stale) Mirror 19: 0x7298c32e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000706 1780336157.249930 write 0x016310 thru 0x017f9f (0x1c90 bytes) 000707 1780336157.270778 trunc from 0x03a8f4 to 0x03cddc (0x24e9 bytes) 000708 1780336157.283088 punch from 0x00e427 to 0x01a1ab (0xbd85 bytes) 000709 1780336157.303535 punch from 0x033c3e to 0x03ffff (0xc3c2 bytes) 000710 1780336157.310537 write 0x03f2b2 thru 0x03ffff (0xd4e bytes) 000711 1780336157.323265 trunc from 0x040000 to 0x00f302 (0x30cfd bytes) 000712 1780336157.343553 trunc from 0x003b98 to 0x004557 (0x9c0 bytes) 000713 1780336157.350280 fallocate 0x01372e thru 0x01e6ac (0xaf7f bytes) 000714 1780336157.357198 read 0x001a38 thru 0x009bf0 (0x81b9 bytes) 000716 1780336157.372307 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xf303): Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf303): Mirror 18: 0xafaae4dc (stale) Mirror 19: 0xafaae4dc (stale) CRC-32 checksum value for chunk [0, 0xf303): Mirror 1: 0x7ede752c Mirror 20: 0x7ede752c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf303: skipped 000717 1780336158.114233 write 0x032679 thru 0x03da76 (0xb3fe bytes) 000719 1780336158.164212 trunc from 0x03da77 to 0x001c87 (0x3bdef bytes) 000720 1780336158.199500 mapread 0x00154e thru 0x001c87 (0x73a bytes) 000721 1780336158.206538 trunc from 0x001c88 to 0x03fd71 (0x3e0ea bytes) 000722 1780336158.226204 mapwrite 0x039267 thru 0x03a8e0 (0x167a bytes) 000723 1780336158.255300 trunc from 0x03ce1b to 0x03fd71 (0x2f57 bytes) 000724 1780336158.261892 write 0x00c01c thru 0x018c7f (0xcc64 bytes) 000725 1780336158.290230 write 0x0041f0 thru 0x00c050 (0x7e61 bytes) 000726 1780336158.300481 punch from 0x005513 to 0x00c75f (0x724d bytes) 000727 1780336158.351487 fallocate 0x00814f thru 0x0081fa (0xac bytes) 000728 1780336158.365490 mapwrite 0x0052dc thru 0x008a91 (0x37b6 bytes) 000729 1780336158.405769 read 0x024788 thru 0x0269db (0x2254 bytes) 000731 1780336158.413818 trunc from 0x03fd72 to 0x004909 (0x3b468 bytes) 000732 1780336158.452263 mapread 0x0007ee thru 0x004410 (0x3c23 bytes) 000733 1780336158.460061 mapwrite 0x0210f8 thru 0x02ee8a (0xdd93 bytes) 000734 1780336158.556584 write 0x01cab3 thru 0x027922 (0xae70 bytes) 000735 1780336158.558423 punch from 0x02c54c to 0x02ee8a (0x293f bytes) 000736 1780336158.566780 trunc from 0x02ee8b to 0x02eec5 (0x3b bytes) 000738 1780336158.595595 trunc from 0x02eec6 to 0x01c56f (0x12956 bytes) 000740 1780336158.627496 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c570): Mirror 18: 0x89969204 (stale) Mirror 20: 0x8eab3eb9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c570): Mirror 18: 0x89969204 (stale) Mirror 20: 0x8eab3eb9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c570: skipped 000741 1780336158.950978 write 0x0058a1 thru 0x011e2a (0xc58a bytes) 000742 1780336158.965597 mapread 0x00133d thru 0x0085e1 (0x72a5 bytes) 000745 1780336158.971721 mapwrite 0x013a92 thru 0x0224db (0xea4a bytes) 000746 1780336159.036289 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x224dc): Mirror 20: 0x39100fd6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x224dc): Mirror 20: 0x39100fd6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x224dc: skipped 000747 1780336159.185628 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x224dc): Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x224dc): Mirror 20: 0x39100fd6 (stale) CRC-32 checksum value for chunk [0, 0x224dc): Mirror 1: 0xe78a25d3 Mirror 21: 0xe78a25d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x224dc: skipped 000748 1780336160.168179 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000749 1780336160.572491 fallocate 0x00876a thru 0x012d75 (0xa60c bytes) 000750 1780336160.648225 mapread 0x01f393 thru 0x0224db (0x3149 bytes) 000751 1780336160.679095 write 0x0107ab thru 0x0199f4 (0x924a bytes) 000753 1780336160.705393 punch from 0x00bae3 to 0x0136d6 (0x7bf4 bytes) 000754 1780336160.742308 trunc from 0x00355a to 0x01153b (0xdfe2 bytes) 000756 1780336160.770622 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000757 1780336161.011028 read 0x01001d thru 0x0115cb (0x15af bytes) 000758 1780336161.017824 mapread 0x0180b3 thru 0x0224db (0xa429 bytes) 000759 1780336161.098787 mapwrite 0x035d44 thru 0x036659 (0x916 bytes) 000760 1780336161.212748 punch from 0x00d9ab to 0x011dff (0x4455 bytes) 000762 1780336161.223947 read 0x008263 thru 0x01307b (0xae19 bytes) 000764 1780336161.263439 mapwrite 0x022f43 thru 0x02dd30 (0xadee bytes) 000768 1780336161.328013 trunc from 0x011beb to 0x01a6d9 (0x8aef bytes) 000769 1780336161.339154 MIRROR_EXTEND to 2 mirrors 000770 1780336162.085827 write 0x031b18 thru 0x0389b4 (0x6e9d bytes) 000771 1780336162.147335 fallocate 0x0205b3 thru 0x02e9b8 (0xe406 bytes) 000772 1780336162.153178 write 0x02938a thru 0x031adb (0x8752 bytes) 000773 1780336162.160205 trunc from 0x0389b5 to 0x02428b (0x14729 bytes) 000774 1780336162.205987 mapwrite 0x017f72 thru 0x018f6d (0xffc bytes) 000775 1780336162.236889 mapwrite 0x018a32 thru 0x01eaa2 (0x6071 bytes) 000776 1780336162.254601 mapwrite 0x024dcc thru 0x02ca0d (0x7c42 bytes) 000777 1780336162.321635 punch from 0x0010f4 to 0x0024e4 (0x13f1 bytes) 000778 1780336162.329311 trunc from 0x02ca0e to 0x02659b (0x6472 bytes) 000779 1780336162.355889 fallocate 0x01063d thru 0x017adc (0x74a0 bytes) 000780 1780336162.363531 trunc from 0x02659c to 0x009751 (0x1ce4a bytes) 000781 1780336162.382142 read 0x004ae5 thru 0x0051a0 (0x6bc bytes) 000782 1780336162.389894 mapread 0x0077ef thru 0x009751 (0x1f63 bytes) 000783 1780336162.404889 trunc from 0x009752 to 0x012929 (0x91d8 bytes) 000785 1780336162.427261 punch from 0x0126df to 0x012929 (0x24b bytes) 000786 1780336162.434404 mapwrite 0x02e284 thru 0x031af1 (0x386e bytes) 000787 1780336162.487344 mapwrite 0x003f3d thru 0x00fba0 (0xbc64 bytes) 000788 1780336162.535315 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000789 1780336162.673772 fallocate 0x03677f thru 0x03b817 (0x5099 bytes) 000790 1780336162.680069 trunc from 0x0301c6 to 0x03b41a (0xb255 bytes) 000791 1780336162.687894 write 0x00abc3 thru 0x00c149 (0x1587 bytes) 000792 1780336162.704634 fallocate 0x01a67e thru 0x01aaba (0x43d bytes) 000794 1780336162.709809 fallocate 0x00857b thru 0x0124c9 (0x9f4f bytes) 000795 1780336162.715007 write 0x039429 thru 0x03ffff (0x6bd7 bytes) 000796 1780336162.724585 fallocate 0x02c01d thru 0x02e12a (0x210e bytes) 000797 1780336162.730679 mapread 0x020459 thru 0x02d9f1 (0xd599 bytes) 000800 1780336162.754863 read 0x021908 thru 0x025051 (0x374a bytes) 000801 1780336162.755952 fallocate 0x00ed77 thru 0x013df5 (0x507f bytes) 000802 1780336162.765295 trunc from 0x004b42 to 0x00c416 (0x78d5 bytes) 000803 1780336162.791952 trunc from 0x026b73 to 0x032dbd (0xc24b bytes) 000804 1780336162.798199 trunc from 0x040000 to 0x005a1c (0x3a5e3 bytes) 000805 1780336162.847243 fallocate 0x018cbf thru 0x0192b4 (0x5f6 bytes) 000806 1780336162.862514 fallocate 0x01cb25 thru 0x028fe7 (0xc4c3 bytes) 000807 1780336162.872600 read 0x01904f thru 0x02247b (0x942d bytes) 000808 1780336162.902316 mapread 0x00598b thru 0x007a7c (0x20f2 bytes) 000809 1780336162.905349 trunc from 0x0158f8 to 0x017468 (0x1b71 bytes) 000810 1780336162.918561 read 0x020921 thru 0x028fe7 (0x86c7 bytes) 000811 1780336162.919342 fallocate 0x03afb6 thru 0x03d37e (0x23c9 bytes) 000812 1780336162.932299 read 0x0394f4 thru 0x03d37e (0x3e8b bytes) 000813 1780336162.949548 mapwrite 0x035306 thru 0x03ffff (0xacfa bytes) 000814 1780336163.016206 trunc from 0x040000 to 0x0364c7 (0x9b38 bytes) 000815 1780336163.111531 trunc from 0x005292 to 0x00bc94 (0x6a03 bytes) 000817 1780336163.120937 write 0x0276a0 thru 0x02fa27 (0x8388 bytes) 000818 1780336163.128609 mapread 0x02b8b5 thru 0x0337e2 (0x7f2e bytes) 000820 1780336163.134349 punch from 0x021db2 to 0x02e6c5 (0xc914 bytes) 000821 1780336163.155179 write 0x035866 thru 0x03f347 (0x9ae2 bytes) 000822 1780336163.158215 trunc from 0x031efe to 0x03f347 (0xd44a bytes) 000823 1780336163.180693 trunc from 0x03f348 to 0x01263b (0x2cd0c bytes) 000824 1780336163.216459 mapwrite 0x034f54 thru 0x03c0db (0x7188 bytes) 000825 1780336163.321123 trunc from 0x03c0dc to 0x02d151 (0xef8a bytes) 000826 1780336163.341304 read 0x011c0f thru 0x012953 (0xd45 bytes) 000827 1780336163.342216 trunc from 0x01bf89 to 0x026ff9 (0xb071 bytes) 000828 1780336163.348285 mapread 0x003761 thru 0x0044bd (0xd5d bytes) 000830 1780336163.351776 punch from 0x023705 to 0x02b12e (0x7a2a bytes) 000831 1780336163.357812 trunc from 0x000e09 to 0x00ac14 (0x9e0c bytes) 000832 1780336163.364638 write 0x0210d6 thru 0x023756 (0x2681 bytes) 000833 1780336163.376036 fallocate 0x036d89 thru 0x03ffff (0x9277 bytes) 000834 1780336163.382853 write 0x02bbc7 thru 0x0343fd (0x8837 bytes) 000835 1780336163.391786 punch from 0x001a7b to 0x007d9f (0x6325 bytes) 000836 1780336163.398031 trunc from 0x040000 to 0x0254ac (0x1ab53 bytes) 000837 1780336163.413635 trunc from 0x0254ad to 0x012308 (0x131a4 bytes) 000838 1780336163.440404 punch from 0x005a1c to 0x00ed88 (0x936d bytes) 000839 1780336163.450381 mapwrite 0x012a09 thru 0x01c083 (0x967b bytes) 000841 1780336163.634226 mapwrite 0x0250bd thru 0x033ebe (0xee02 bytes) 000842 1780336163.761733 mapwrite 0x018205 thru 0x0238c2 (0xb6be bytes) 000843 1780336163.890399 punch from 0x005afd to 0x014504 (0xea08 bytes) 000844 1780336163.896922 punch from 0x00e31f to 0x014df0 (0x6ad2 bytes) 000845 1780336163.904494 trunc from 0x01551a to 0x021a6d (0xc554 bytes) 000846 1780336163.917695 read 0x03227b thru 0x033ebe (0x1c44 bytes) 000848 1780336163.919747 mapwrite 0x022b07 thru 0x025257 (0x2751 bytes) 000849 1780336163.969989 mapread 0x01f549 thru 0x028e5a (0x9912 bytes) 000850 1780336163.972211 punch from 0x0163d5 to 0x01739b (0xfc7 bytes) 000851 1780336163.980043 write 0x029986 thru 0x033fb6 (0xa631 bytes) 000852 1780336163.990433 MIRROR_EXTEND to 2 mirrors 000853 1780336165.019218 mapwrite 0x01054c thru 0x016138 (0x5bed bytes) 000854 1780336165.161150 mapread 0x022c9e thru 0x02eb86 (0xbee9 bytes) 000855 1780336165.184296 fallocate 0x01e333 thru 0x020d6a (0x2a38 bytes) 000856 1780336165.192086 trunc from 0x0253eb to 0x033fb6 (0xebcc bytes) 000857 1780336165.201736 mapread 0x005875 thru 0x010298 (0xaa24 bytes) 000858 1780336165.214621 read 0x02eadc thru 0x030a61 (0x1f86 bytes) 000859 1780336165.223391 mapread 0x000bf4 thru 0x00256d (0x197a bytes) 000860 1780336165.231389 mapwrite 0x00e1a2 thru 0x01096d (0x27cc bytes) 000861 1780336165.247735 trunc from 0x033fb7 to 0x004261 (0x2fd55 bytes) 000862 1780336165.284757 trunc from 0x00274f to 0x004261 (0x1b13 bytes) 000863 1780336165.290901 mapread 0x001027 thru 0x004261 (0x323b bytes) 000865 1780336165.305106 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x4262): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4262): Mirror 2: 0x479938da (stale) CRC-32 checksum value for chunk [0, 0x4262): Mirror 1: 0x479938da Mirror 3: 0x479938da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4262: skipped 000866 1780336166.685365 write 0x03c85c thru 0x03ffff (0x37a4 bytes) 000867 1780336166.752597 write 0x035f10 thru 0x03a421 (0x4512 bytes) 000868 1780336166.777260 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xcf68c613 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xcf68c613 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000870 1780336167.372819 fallocate 0x01873c thru 0x01aead (0x2772 bytes) 000871 1780336167.393365 trunc from 0x03a56f to 0x03ffff (0x5a91 bytes) 000872 1780336167.409155 read 0x0245f4 thru 0x0297f9 (0x5206 bytes) 000873 1780336167.422798 read 0x031ceb thru 0x0343ef (0x2705 bytes) 000874 1780336167.446586 write 0x020299 thru 0x02eac3 (0xe82b bytes) 000875 1780336167.463721 write 0x008d4a thru 0x016f3e (0xe1f5 bytes) 000876 1780336167.470718 punch from 0x025048 to 0x02f9ba (0xa973 bytes) 000877 1780336167.507320 mapread 0x02a8a3 thru 0x034b87 (0xa2e5 bytes) 000878 1780336167.606045 read 0x002e59 thru 0x00f6f0 (0xc898 bytes) 000879 1780336167.606830 mapread 0x0321c3 thru 0x032acc (0x90a bytes) 000880 1780336167.612526 mapwrite 0x03767c thru 0x03e2ee (0x6c73 bytes) 000881 1780336167.666412 write 0x03cf19 thru 0x03e52a (0x1612 bytes) 000883 1780336167.671695 mapwrite 0x00f818 thru 0x011d6d (0x2556 bytes) 000885 1780336167.705911 fallocate 0x0305cf thru 0x035fd5 (0x5a07 bytes) 000886 1780336167.711509 trunc from 0x040000 to 0x03a337 (0x5cc8 bytes) 000887 1780336167.774587 mapread 0x0160af thru 0x0212b7 (0xb209 bytes) 000888 1780336167.780477 trunc from 0x03a338 to 0x031868 (0x8acf bytes) 000889 1780336167.829137 trunc from 0x02ab95 to 0x031868 (0x6cd4 bytes) 000890 1780336167.835030 trunc from 0x031869 to 0x020e99 (0x109cf bytes) 000891 1780336167.850405 write 0x00a985 thru 0x012644 (0x7cc0 bytes) 000892 1780336167.853227 mapread 0x020712 thru 0x020e99 (0x788 bytes) 000893 1780336167.854398 fallocate 0x01249c thru 0x01faaa (0xd60f bytes) 000894 1780336167.861149 trunc from 0x020e9a to 0x0049eb (0x1c4ae bytes) 000896 1780336167.878485 fallocate 0x02569b thru 0x033a57 (0xe3bd bytes) 000897 1780336167.885293 trunc from 0x033a58 to 0x0206b6 (0x133a1 bytes) 000898 1780336167.906051 trunc from 0x004453 to 0x00578a (0x1338 bytes) 000899 1780336167.911350 punch from 0x014d95 to 0x0206b6 (0xb922 bytes) 000900 1780336167.917591 punch from 0x0200dc to 0x0206b6 (0x5db bytes) 000901 1780336167.923191 fallocate 0x00a29f thru 0x00e715 (0x4477 bytes) 000902 1780336167.934126 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000903 1780336168.075261 trunc from 0x015d60 to 0x0206b6 (0xa957 bytes) 000904 1780336168.083126 punch from 0x01c040 to 0x0206b6 (0x4677 bytes) 000905 1780336168.088356 fallocate 0x039d12 thru 0x03c8dc (0x2bcb bytes) 000906 1780336168.094691 write 0x00bc61 thru 0x0135ed (0x798d bytes) 000907 1780336168.117707 read 0x0151fa thru 0x022466 (0xd26d bytes) 000908 1780336168.127318 punch from 0x0159a4 to 0x0209cc (0xb029 bytes) 000909 1780336168.132732 trunc from 0x0184c3 to 0x01ea10 (0x654e bytes) 000911 1780336168.142640 write 0x02bf17 thru 0x0361c3 (0xa2ad bytes) 000912 1780336168.157299 trunc from 0x00381f to 0x00eb39 (0xb31b bytes) 000913 1780336168.173560 mapwrite 0x016099 thru 0x01b9b5 (0x591d bytes) 000915 1780336168.211799 write 0x027283 thru 0x0349ab (0xd729 bytes) 000916 1780336168.214774 read 0x032c02 thru 0x03c8dc (0x9cdb bytes) 000917 1780336168.226458 mapwrite 0x0129dc thru 0x0178a3 (0x4ec8 bytes) 000918 1780336168.245712 trunc from 0x03c8dd to 0x028eca (0x13a12 bytes) 000919 1780336168.261371 punch from 0x0032ee to 0x004997 (0x16aa bytes) 000920 1780336168.268582 trunc from 0x028ecb to 0x0004ca (0x28a00 bytes) 000921 1780336168.294821 trunc from 0x0004cb to 0x005910 (0x5446 bytes) 000922 1780336168.336158 trunc from 0x000aa8 to 0x005910 (0x4e69 bytes) 000923 1780336168.344712 punch from 0x000f35 to 0x005910 (0x49dc bytes) 000925 1780336168.355761 punch from 0x00573f to 0x005910 (0x1d2 bytes) 000926 1780336168.360230 trunc from 0x005911 to 0x0086d1 (0x2dc1 bytes) 000927 1780336168.386505 trunc from 0x00697c to 0x0086d1 (0x1d56 bytes) 000928 1780336168.392288 fallocate 0x03ea69 thru 0x03ffff (0x1597 bytes) 000929 1780336168.397353 trunc from 0x0086d2 to 0x038cd4 (0x30603 bytes) 000930 1780336168.412292 read 0x01dbca thru 0x01f85d (0x1c94 bytes) 000932 1780336168.427139 write 0x01aa0e thru 0x02445f (0x9a52 bytes) 000933 1780336168.429218 read 0x02bb63 thru 0x036437 (0xa8d5 bytes) 000934 1780336168.430265 trunc from 0x026379 to 0x0312e4 (0xaf6c bytes) 000935 1780336168.435640 mapread 0x001f5b thru 0x00fcfb (0xdda1 bytes) 000937 1780336168.439161 fallocate 0x0261dd thru 0x0273ff (0x1223 bytes) 000938 1780336168.444497 read 0x0135e7 thru 0x021696 (0xe0b0 bytes) 000939 1780336168.445625 trunc from 0x038cd5 to 0x03bace (0x2dfa bytes) 000940 1780336168.466292 punch from 0x00c3bc to 0x017200 (0xae45 bytes) 000942 1780336168.474456 read 0x00ba69 thru 0x017a2e (0xbfc6 bytes) 000943 1780336168.484762 write 0x030919 thru 0x036f3e (0x6626 bytes) 000944 1780336168.489139 punch from 0x0333cc to 0x03bace (0x8703 bytes) 000945 1780336168.524609 punch from 0x03b2f4 to 0x03bace (0x7db bytes) 000946 1780336168.530558 trunc from 0x03bacf to 0x029a84 (0x1204a bytes) 000947 1780336168.549573 trunc from 0x026d96 to 0x029a84 (0x2cef bytes) 000948 1780336168.555895 punch from 0x00801d to 0x0119eb (0x99cf bytes) 000949 1780336168.561271 trunc from 0x029a85 to 0x01b68f (0xe3f5 bytes) 000950 1780336168.586586 mapread 0x0050e8 thru 0x007644 (0x255d bytes) 000951 1780336168.594234 MIRROR_EXTEND to 2 mirrors 000952 1780336169.559125 trunc from 0x0072d1 to 0x00839d (0x10cd bytes) 000953 1780336169.642559 write 0x038321 thru 0x03ffff (0x7cdf bytes) 000954 1780336169.664928 mapread 0x007f25 thru 0x00f320 (0x73fc bytes) 000955 1780336169.714665 write 0x01cdfe thru 0x0272fd (0xa500 bytes) 000956 1780336169.784454 MIRROR_RESYNC 2 mirrors 000957 1780336170.865674 mapwrite 0x03131a thru 0x0319ce (0x6b5 bytes) 000958 1780336170.958914 fallocate 0x03493d thru 0x03d7cf (0x8e93 bytes) 000959 1780336170.970459 fallocate 0x009f24 thru 0x014a97 (0xab74 bytes) 000960 1780336171.001800 MIRROR_RESYNC 2 mirrors 000961 1780336171.987245 mapread 0x012881 thru 0x01b9f6 (0x9176 bytes) 000962 1780336172.023627 mapread 0x0302d8 thru 0x038fd6 (0x8cff bytes) 000963 1780336172.044969 MIRROR_EXTEND to 3 mirrors 000964 1780336172.919491 write 0x003075 thru 0x00c610 (0x959c bytes) 000965 1780336172.965924 mapwrite 0x023d9f thru 0x0316be (0xd920 bytes) 000966 1780336172.995070 write 0x01562e thru 0x024898 (0xf26b bytes) 000967 1780336173.003490 write 0x02f9bd thru 0x035814 (0x5e58 bytes) 000968 1780336173.018457 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfadd0e9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfadd0e9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000969 1780336173.580441 trunc from 0x008a84 to 0x00d563 (0x4ae0 bytes) 000970 1780336173.596803 mapwrite 0x021caa thru 0x03073f (0xea96 bytes) 000971 1780336173.713718 punch from 0x000851 to 0x0078eb (0x709b bytes) 000972 1780336173.731028 trunc from 0x03e427 to 0x03e943 (0x51d bytes) 000973 1780336173.751094 punch from 0x030984 to 0x030ef3 (0x570 bytes) 000974 1780336173.758166 trunc from 0x040000 to 0x03a273 (0x5d8c bytes) 000975 1780336173.800476 fallocate 0x006987 thru 0x00cece (0x6548 bytes) 000976 1780336173.809945 punch from 0x02caad to 0x031bc1 (0x5115 bytes) 000977 1780336173.831388 trunc from 0x03538a to 0x03a273 (0x4eea bytes) 000978 1780336173.842290 trunc from 0x028ee4 to 0x02b5d7 (0x26f4 bytes) 000979 1780336173.877530 mapread 0x00589f thru 0x0095aa (0x3d0c bytes) 000980 1780336173.895581 trunc from 0x031439 to 0x03a273 (0x8e3b bytes) 000981 1780336173.904951 write 0x02b0fd thru 0x02cc68 (0x1b6c bytes) 000983 1780336173.921678 mapread 0x011990 thru 0x016061 (0x46d2 bytes) 000984 1780336173.955082 trunc from 0x03a274 to 0x00e94e (0x2b925 bytes) 000986 1780336173.996999 read 0x009058 thru 0x00e94e (0x58f7 bytes) 000987 1780336173.998632 mapwrite 0x02e4c2 thru 0x02ea48 (0x587 bytes) 000988 1780336174.074309 trunc from 0x02d79e to 0x02ea48 (0x12ab bytes) 000990 1780336174.107627 trunc from 0x004995 to 0x009682 (0x4cee bytes) 000991 1780336174.114445 mapwrite 0x037d7b thru 0x03b79b (0x3a21 bytes) 000992 1780336174.232006 mapread 0x021480 thru 0x02eac4 (0xd645 bytes) 000993 1780336174.291198 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b79c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b79c): Mirror 2: 0x29a7f572 (stale) CRC-32 checksum value for chunk [0, 0x3b79c): Mirror 1: 0xadbab36d Mirror 3: 0xadbab36d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b79c: skipped 000994 1780336175.543954 fallocate 0x01ad6c thru 0x0285ef (0xd884 bytes) 000995 1780336175.640714 mapwrite 0x03e07f thru 0x03ffff (0x1f81 bytes) 000996 1780336175.782646 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfadd0e9d (stale) Mirror 3: 0x69cee04a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe5cb6cb9 Mirror 4: 0xe5cb6cb9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000997 1780336177.086607 write 0x00777d thru 0x00849a (0xd1e bytes) 000998 1780336177.171634 read 0x02a97d thru 0x036f6f (0xc5f3 bytes) 000999 1780336177.184212 read 0x0319ba thru 0x03a6d9 (0x8d20 bytes) 001000 1780336177.200660 trunc from 0x040000 to 0x015440 (0x2abbf bytes) All operations completed A-OK! PASS 70a (108s) == sanity-flr test 71: check mirror extend parent fid ==== 13:49:45 (1780336185) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x1a9:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 13:49:58 (1780336198) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16329 s, 12.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141799 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e2:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 13:50:09 (1780336209) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre2 Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1277344 .. done Extending file size to 6975744 .. Extending file size to 1620832 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8125664 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8671328 .. Extending file size to 8697408 .. Extending file size to 8566784 .. Extending file size to 8801952 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9020608 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9070496 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9062848 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 60917 60918 60919 60920 60921 failed 192.168.201.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg143-client.virtnet /mnt/lustre2 (opts:) 192.168.201.143@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg143-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:51:29 (1780336289) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre2 Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1774464 .. Extending file size to 1962240 .. failed Extending file size to 5042112 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5386176 .. Extending file size to 6819072 .. Extending file size to 8550208 .. Extending file size to 6605760 .. Extending file size to 8075904 .. Extending file size to 8912960 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9266880 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 62556 62558 62559 62560 62561 192.168.201.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg143-client.virtnet /mnt/lustre2 (opts:) 192.168.201.143@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg143-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:52:47 (1780336367) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2456 1268848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1996 1269308 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24796 3582224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18960 3588060 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43756 7170284 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2456 1268848 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1996 1269308 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24796 3582224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18960 3588060 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43756 7170284 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110466 s, 19.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.296908 s, 7.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x8ef5f436 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg143-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:53:10 (1780336390) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102117 s, 20.5 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 13:53:18 (1780336398) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d9:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x2f7:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:53:26 (1780336406) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0994885 s, 21.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:53:34 (1780336414) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0919766 s, 22.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2db:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2db:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:53:42 (1780336422) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140491 s, 14.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fb:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:54:16 (1780336456) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg143-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616321 s, 17.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x50:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x50:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } PASS 204d (51s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:55:08 (1780336508) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg143-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.212732 s, 9.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:56:03 (1780336563) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg143-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg143-server: Pool lustre.test_204f created Adding targets to pool oleg143-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg143-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132715 s, 15.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg143-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg143-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg143-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:57:18 (1780336638) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:57:27 (1780336647) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:57:34 (1780336654) oleg143-server: Pool lustre.test_206 created Adding targets to pool oleg143-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg143-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg143-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg143-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg143-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:58:00 (1780336680) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105277 s, 10.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382681 s, 27.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xfe8591f4 Mirror 3: 0xfe8591f4 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:58:10 (1780336690) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:58:14 (1780336694) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:58:18 (1780336698) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0516654 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb10000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x6d0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x6d0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x6d0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x910000 Read this (4 bytes): BBBB 192.168.201.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg143-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:59:00 (1780336740) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0433499 s, 189 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:59:08 (1780336748) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396238 s, 26.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1d9 lmm_fid: [0x200000405:0x1d9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1d9 lmm_fid: [0x200000405:0x1d9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e8:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611609 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 13:59:09.000000000 -0400 Modify: 2026-06-01 13:59:09.000000000 -0400 Change: 2026-06-01 13:59:11.000000000 -0400 Birth: 2026-06-01 13:59:08.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:59:18 (1780336758) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477643 s, 22.0 MB/s oleg143-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg143-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg143-server: O/280000401/d9/309: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:59:40 (1780336780) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.154663 s, 265 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0367394 s, 111 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 1 13:59 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1de:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 1 13:59 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 13:59:49 (1780336789) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 13:59:58 (1780336798) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1915 sec =========== 14:00:07 (1780336807) === sanity-flr: start cleanup 14:00:09 (1780336809) === === sanity-flr: finish cleanup 14:00:18 (1780336818) ===