-----============= acceptance-small: sanity-flr ============----- Mon May 18 13:39:15 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg636-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 13:39:32 (1779125972) === oleg636-client.virtnet: executing check_config_client /mnt/lustre oleg636-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg636-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9784c775d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9784c775d800.idle_timeout=debug disable quota as required oleg636-server: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 13:40:07 (1779126007) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8227] == sanity-flr test 0a: lfs mirror create with -N option == 13:40:09 (1779126009) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:40:24 (1779126024) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:40:27 (1779126027) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:40:29 (1779126029) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:40:48 (1779126048) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:40:50 (1779126050) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 13:40:53 (1779126053) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9784c775d800.stats=clear osc.lustre-OST0001-osc-ffff9784c775d800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:41:00 (1779126060) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:41:09 (1779126069) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:41:17 (1779126077) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:41:20 (1779126080) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:41:28 (1779126088) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 13:41:37 (1779126097) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:41:44 (1779126104) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:41:52 (1779126112) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:42:01 (1779126121) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:42:11 (1779126131) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:42:18 (1779126138) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.732391 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698533 s, 15.0 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:42:28 (1779126148) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.397911 s, 26.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:42:36 (1779126156) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:42:47 (1779126167) Stopping /mnt/lustre-ost1 (opts:) on oleg636-server oleg636-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9784c775d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9784c775d800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg636-server: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg636-server: oleg636-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg636-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg636-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg636-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg636-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg636-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg636-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg636-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg636-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg636-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg636-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg636-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg636-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg636-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg636-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg636-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg636-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg636-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg636-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg636-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg636-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg636-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg636-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg636-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg636-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg636-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg636-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg636-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg636-server oleg636-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9784c775d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9784c775d800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg636-server: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg636-server: oleg636-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg636-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (131s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:44:58 (1779126298) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.173269 s, 18.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg636-server oleg636-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9784c775d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9784c775d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg636-server: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg636-server: oleg636-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg636-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:45:39 (1779126339) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg636-server oleg636-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9784c775d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9784c775d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg636-server: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg636-server: oleg636-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg636-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg636-server oleg636-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9784c775d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9784c775d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg636-server: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg636-server: oleg636-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg636-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (67s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:46:45 (1779126405) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.8504 s, 18.4 MB/s -rw-r--r-- 1 root root 52428800 May 18 13:46 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.924s user 0m0.000s sys 0m0.168s Stopping /mnt/lustre-ost1 (opts:) on oleg636-server oleg636-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9784c775d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9784c775d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.339s user 0m0.000s sys 0m0.225s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg636-server: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg636-server: oleg636-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg636-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg636-server oleg636-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9784c775d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9784c775d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.110s user 0m0.000s sys 0m0.164s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg636-server: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg636-server: oleg636-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg636-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg636-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9784c775d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9784c775d800.ost_server_uuid in FULL state after 0 sec PASS 33b (84s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:48:09 (1779126489) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:48:12 (1779126492) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:48:16 (1779126496) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 13:48:20 (1779126500) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 13:48:27 (1779126507) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.562145 s, 22.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:48:39 (1779126519) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510718 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870315 s, 12.0 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:48:48 (1779126528) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:48:56 (1779126536) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00305398 s, 335 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000760913 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00312622 s, 328 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00121063 s, 846 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 13:49:06 (1779126546) create test files with size 15744627 .. 1058b51ed7f6ea62f31aa8efc306f4b7 - 3d9cb4db847c73e2370612ac40700d98 - 8f989d0c5a54ab69ec808b11c6072d79 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg636-client.virtnet /mnt/lustre (opts:) Starting client: oleg636-client.virtnet: -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 13:49:30 (1779126570) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:49:45 (1779126585) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:49:51 (1779126591) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823328 s, 12.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376128 s, 27.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 13:50:00 (1779126600) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132959 s, 15.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.393131 s, 10.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17204 s, 12.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 13:50:14 (1779126614) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 13:50:17 (1779126617) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:50:20 (1779126620) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399953 s, 26.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 13:50:28 (1779126628) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.365027 s, 8.6 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:50:37 (1779126637) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:50:47 (1779126647) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.429879 s, 24.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg636-client.virtnet /mnt/lustre (opts:) Starting client: oleg636-client.virtnet: -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:50:57 (1779126657) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9784e0f7d800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.27745 s, 37.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg636-client.virtnet /mnt/lustre (opts:) Starting client: oleg636-client.virtnet: -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9784da835800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:51:11 (1779126671) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:51:56 (1779126716) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:52:29 (1779126749) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 13:52:32 (1779126752) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.23022 s, 13.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.162741 s, 19.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x3c1ce0a3 (nosync,stale) Mirror 2: 0x3c1ce0a3 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2b175a1b8bbaa06bf9ab489a88429d9c - mirror 1 checksum: 95756118c0f053fa74f3458bc25ba1f7 - mirror 2 checksum: 2b175a1b8bbaa06bf9ab489a88429d9c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon May 18 13:52:34 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon May 18 13:52:34 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2b175a1b8bbaa06bf9ab489a88429d9c - mirror 1 checksum: 95756118c0f053fa74f3458bc25ba1f7 - mirror 2 checksum: 95756118c0f053fa74f3458bc25ba1f7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon May 18 13:52:34 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon May 18 13:52:34 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 13:52:42 (1779126762) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543723 s, 19.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:52:49 (1779126769) Failing mds1 on oleg636-server Stopping /mnt/lustre-mds1 (opts:) on oleg636-server 13:53:01 (1779126781) shut down facet: mds1 facet_host: oleg636-server facet_failover_host: oleg636-server Failover mds1 to oleg636-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg636-server: oleg636-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Started lustre-MDT0000 13:53:33 (1779126813) targets are mounted 13:53:33 (1779126813) facet_failover done oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340215 s, 30.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 50A (62s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:53:51 (1779126831) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0542433 s, 378 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0234514 s, 873 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:54:03 (1779126843) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0610866 s, 335 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0178315 s, 1.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:54:14 (1779126854) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00516873 s, 3.2 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:54:22 (1779126862) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0745598 s, 275 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:54:31 (1779126871) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0118991 s, 344 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0310061 s, 132 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:54:38 (1779126878) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:55:04 (1779126904) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:55:32 (1779126932) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:55:57 (1779126957) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:56:22 (1779126982) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 13:56:29 (1779126989) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57189 mirror split pid 57191 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 57189 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 57191 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 13:57:37 (1779127057) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 7059 skipping zero length punch hole 000002 1779127059.448821 mapwrite 0x01bc09 thru 0x02acbd (0xf0b5 bytes) fallocating to largest ever: 0x40000 000003 1779127059.592384 fallocate 0x03cfb8 thru 0x03ffff (0x3048 bytes) 000004 1779127059.604679 trunc from 0x02acbe to 0x00e581 (0x1c73c bytes) 000005 1779127059.636722 mapread 0x002022 thru 0x004cc5 (0x2ca4 bytes) 000006 1779127059.685275 punch from 0x003a51 to 0x00d802 (0x9db2 bytes) 000007 1779127059.708836 read 0x00c04a thru 0x00cdd1 (0xd88 bytes) 000008 1779127059.715815 read 0x00c63a thru 0x00e581 (0x1f48 bytes) 000009 1779127059.733341 mapwrite 0x0113a6 thru 0x0207bc (0xf417 bytes) 000010 1779127059.894876 write 0x008176 thru 0x0113cc (0x9257 bytes) 000012 1779127059.897617 trunc from 0x0207bd to 0x01ab2c (0x5c90 bytes) 000013 1779127059.947565 mapread 0x00a9a3 thru 0x01147f (0x6add bytes) 000014 1779127059.950906 mapwrite 0x016131 thru 0x01cfb0 (0x6e80 bytes) 000015 1779127060.059160 punch from 0x018729 to 0x01cfb0 (0x4888 bytes) 000018 1779127060.069199 fallocate 0x02ec69 thru 0x03e010 (0xf3a8 bytes) 000019 1779127060.076772 fallocate 0x00478d thru 0x00bf26 (0x779a bytes) 000020 1779127060.085513 write 0x037cae thru 0x03ffff (0x8352 bytes) 000021 1779127060.089839 trunc from 0x040000 to 0x01cf42 (0x230bd bytes) 000022 1779127060.124123 write 0x039510 thru 0x03cdf3 (0x38e4 bytes) 000023 1779127060.125447 trunc from 0x03cdf4 to 0x0069c5 (0x3642e bytes) 000024 1779127060.155607 trunc from 0x0069c6 to 0x00173d (0x5288 bytes) 000026 1779127060.190450 write 0x006217 thru 0x00d052 (0x6e3c bytes) 000027 1779127060.199575 read 0x00af5e thru 0x00d052 (0x20f5 bytes) 000028 1779127060.200977 punch from 0x0083f2 to 0x009535 (0x1144 bytes) 000030 1779127060.247574 fallocate 0x0197aa thru 0x02825f (0xeab6 bytes) 000031 1779127060.258426 trunc from 0x028260 to 0x0007e3 (0x27a7c bytes) 000032 1779127060.298887 trunc from 0x000066 to 0x0007e3 (0x77e bytes) 000033 1779127060.310149 trunc from 0x00058c to 0x0007e3 (0x258 bytes) 000034 1779127060.332009 punch from 0x00032a to 0x0007e3 (0x4ba bytes) 000035 1779127060.344753 read 0x0007d0 thru 0x0007e3 (0x14 bytes) 000036 1779127060.377811 read 0x000329 thru 0x0007e3 (0x4bb bytes) 000038 1779127060.391398 MIRROR_EXTEND to 2 mirrors 000039 1779127061.155086 MIRROR_RESYNC 2 mirrors 000040 1779127061.822788 trunc from 00000000 to 0x0007e3 (0x7e4 bytes) 000041 1779127061.947312 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000042 1779127062.214589 trunc from 0x0007e4 to 0x0127a1 (0x11fbe bytes) 000043 1779127062.325375 trunc from 0x0127a2 to 0x013e9b (0x16fa bytes) 000045 1779127062.373848 read 0x012c3d thru 0x013e9b (0x125f bytes) 000046 1779127062.387593 fallocate 0x034fda thru 0x03ffff (0xb026 bytes) 000047 1779127062.395794 read 0x00d39c thru 0x010df0 (0x3a55 bytes) 000048 1779127062.427189 mapwrite 0x037c2b thru 0x03bc82 (0x4058 bytes) 000049 1779127062.468303 mapwrite 0x0058c0 thru 0x00a12f (0x4870 bytes) 000052 1779127062.497760 trunc from 0x040000 to 0x0206c5 (0x1f93a bytes) 000054 1779127062.577171 trunc from 0x0206c6 to 0x00468d (0x1c038 bytes) 000055 1779127062.638217 trunc from 0x0007c9 to 0x00468d (0x3ec5 bytes) 000057 1779127062.677864 MIRROR_EXTEND to 2 mirrors 000058 1779127063.783864 mapread 0x003b1b thru 0x00468d (0xb73 bytes) 000059 1779127063.800387 read 0x001545 thru 0x00468d (0x3149 bytes) 000060 1779127063.810490 read 0x00272c thru 0x00468d (0x1f62 bytes) 000061 1779127063.811353 trunc from 0x00468e to 0x02bc7f (0x275f2 bytes) 000062 1779127063.889123 trunc from 0x02bc80 to 0x007ae1 (0x2419e bytes) 000063 1779127063.922678 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000064 1779127064.134368 mapwrite 0x027ca1 thru 0x029652 (0x19b2 bytes) 000065 1779127064.232058 trunc from 0x0104d4 to 0x01e604 (0xe131 bytes) 000067 1779127064.239494 trunc from 0x015710 to 0x016154 (0xa45 bytes) 000068 1779127064.244538 trunc from 0x029653 to 0x02f777 (0x6125 bytes) 000069 1779127064.260569 trunc from 0x024d9b to 0x02c598 (0x77fe bytes) 000070 1779127064.267289 write 0x03373d thru 0x03554a (0x1e0e bytes) 000071 1779127064.270656 trunc from 0x03554b to 0x03f188 (0x9c3e bytes) 000072 1779127064.288125 punch from 0x038c9e to 0x03e16e (0x54d1 bytes) 000074 1779127064.295499 punch from 0x02ef78 to 0x0339b7 (0x4a40 bytes) 000075 1779127064.304139 trunc from 0x01e68b to 0x02c02a (0xd9a0 bytes) 000077 1779127064.310608 trunc from 0x03f189 to 0x02263d (0x1cb4b bytes) 000078 1779127064.332507 read 0x008233 thru 0x0127a6 (0xa574 bytes) 000079 1779127064.350629 write 0x00ced1 thru 0x017112 (0xa242 bytes) 000080 1779127064.365729 trunc from 0x01394d to 0x01af72 (0x7626 bytes) 000082 1779127064.396246 punch from 0x00945b to 0x01054b (0x70f1 bytes) 000084 1779127064.409584 trunc from 0x00e098 to 0x0134ac (0x5415 bytes) 000085 1779127064.417882 mapwrite 0x013d08 thru 0x01dfc1 (0xa2ba bytes) 000086 1779127064.499435 MIRROR_EXTEND to 2 mirrors 000087 1779127065.371816 punch from 0x020b04 to 0x02263d (0x1b3a bytes) 000089 1779127065.453970 fallocate 0x0034af thru 0x009f21 (0x6a73 bytes) 000090 1779127065.465386 punch from 0x016a90 to 0x02263d (0xbbae bytes) 000091 1779127065.475355 write 0x021411 thru 0x02e89f (0xd48f bytes) 000092 1779127065.491205 punch from 0x024fb0 to 0x029209 (0x425a bytes) 000093 1779127065.507486 punch from 0x01de9c to 0x021757 (0x38bc bytes) 000094 1779127065.513546 mapread 0x01dcb1 thru 0x020214 (0x2564 bytes) 000095 1779127065.571604 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000096 1779127065.780221 read 0x02a34b thru 0x02e89f (0x4555 bytes) 000097 1779127065.817719 write 0x006255 thru 0x007fe1 (0x1d8d bytes) 000098 1779127065.842663 punch from 0x02258d to 0x02e89f (0xc313 bytes) 000099 1779127065.850729 read 0x024cf1 thru 0x025c4a (0xf5a bytes) 000100 1779127065.870679 trunc from 0x02b5a3 to 0x02e89f (0x32fd bytes) 000101 1779127065.877559 trunc from 0x00ff4e to 0x01bf80 (0xc033 bytes) 000102 1779127065.883012 write 0x029afd thru 0x039767 (0xfc6b bytes) 000103 1779127065.887694 mapread 0x016a6a thru 0x017a5f (0xff6 bytes) 000104 1779127065.908694 mapread 0x0016ec thru 0x001d57 (0x66c bytes) 000105 1779127065.909805 read 0x013f8d thru 0x019571 (0x55e5 bytes) 000106 1779127065.942570 fallocate 0x00a951 thru 0x00fef1 (0x55a1 bytes) 000107 1779127065.953668 mapwrite 0x0217ad thru 0x0307f2 (0xf046 bytes) 000108 1779127066.008462 mapread 0x002211 thru 0x00690b (0x46fb bytes) 000109 1779127066.010455 punch from 0x01e4f6 to 0x02c9df (0xe4ea bytes) 000110 1779127066.018688 fallocate 0x02ea78 thru 0x03c2a8 (0xd831 bytes) 000111 1779127066.029194 write 0x00150f thru 0x00bcb7 (0xa7a9 bytes) 000112 1779127066.030760 mapread 0x014988 thru 0x020e21 (0xc49a bytes) 000115 1779127066.055566 trunc from 0x039768 to 0x008b52 (0x30c15 bytes) 000116 1779127066.087632 fallocate 0x02c49e thru 0x03964e (0xd1b1 bytes) 000118 1779127066.096599 mapread 0x02c062 thru 0x02e0a0 (0x203f bytes) 000119 1779127066.117534 trunc from 0x03964f to 0x01182f (0x27e1f bytes) 000120 1779127066.136189 punch from 0x004c5d to 0x00ec5f (0xa003 bytes) 000121 1779127066.155686 punch from 0x0103db to 0x01182f (0x1455 bytes) 000122 1779127066.162239 mapwrite 0x03a7f8 thru 0x03ffff (0x5808 bytes) 000123 1779127066.235563 punch from 0x00a308 to 0x00f457 (0x5150 bytes) 000124 1779127066.239865 trunc from 0x040000 to 0x030e40 (0xf1bf bytes) 000127 1779127066.269868 punch from 0x0266b0 to 0x02cae8 (0x6439 bytes) 000128 1779127066.275393 fallocate 0x01337e thru 0x016183 (0x2e06 bytes) 000129 1779127066.282716 trunc from 0x005ac5 to 0x013ee4 (0xe420 bytes) 000130 1779127066.295404 read 0x02b6e7 thru 0x030e40 (0x575a bytes) 000131 1779127066.310281 punch from 0x00bd8e to 0x017e21 (0xc094 bytes) 000132 1779127066.319645 fallocate 0x02d82f thru 0x03137d (0x3b4f bytes) 000133 1779127066.328835 write 0x021fe9 thru 0x0263b9 (0x43d1 bytes) 000134 1779127066.330439 mapread 0x02bee7 thru 0x02ec35 (0x2d4f bytes) 000136 1779127066.333285 write 0x0128e1 thru 0x01cfb9 (0xa6d9 bytes) 000137 1779127066.352706 trunc from 0x030dfb to 0x03137d (0x583 bytes) 000138 1779127066.357876 trunc from 0x02afc1 to 0x03137d (0x63bd bytes) 000139 1779127066.362640 write 0x032e6f thru 0x03ffff (0xd191 bytes) 000141 1779127066.364999 fallocate 0x002f9f thru 0x00f921 (0xc983 bytes) 000142 1779127066.370411 fallocate 0x03c193 thru 0x03ffff (0x3e6d bytes) 000144 1779127066.377745 read 0x01b760 thru 0x021fa3 (0x6844 bytes) 000145 1779127066.378484 read 0x038efa thru 0x03ffff (0x7106 bytes) 000146 1779127066.380081 mapwrite 0x032957 thru 0x0343c5 (0x1a6f bytes) 000147 1779127066.411769 read 0x02bec9 thru 0x03a7af (0xe8e7 bytes) 000148 1779127066.422003 trunc from 0x040000 to 0x01b2de (0x24d21 bytes) 000149 1779127066.461221 MIRROR_EXTEND to 2 mirrors 000151 1779127067.640279 mapwrite 0x03a3c3 thru 0x03ee71 (0x4aaf bytes) 000152 1779127067.746719 trunc from 0x00fa44 to 0x014a5f (0x501c bytes) 000153 1779127067.753537 trunc from 0x01174e to 0x014679 (0x2f2c bytes) 000154 1779127067.767495 write 0x0378b2 thru 0x03e39c (0x6aeb bytes) 000155 1779127067.780189 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ee72): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee72): Mirror 2: 0x78d0c051 (stale) CRC-32 checksum value for chunk [0, 0x3ee72): Mirror 1: 0x6c7218c0 Mirror 3: 0x6c7218c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee72: skipped 000156 1779127068.904048 mapwrite 0x01cab6 thru 0x02a959 (0xdea4 bytes) 000157 1779127069.090961 mapread 0x00845f thru 0x014dd7 (0xc979 bytes) 000158 1779127069.124012 mapwrite 0x01a637 thru 0x01eebf (0x4889 bytes) 000159 1779127069.188773 punch from 0x006928 to 0x012f6b (0xc644 bytes) 000160 1779127069.204666 write 0x025b92 thru 0x03010d (0xa57c bytes) 000161 1779127069.262182 trunc from 0x00c950 to 0x01a7b0 (0xde61 bytes) 000163 1779127069.310166 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ee72): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee72): Mirror 2: 0x78d0c051 (stale) Mirror 3: 0x6c7218c0 (stale) CRC-32 checksum value for chunk [0, 0x3ee72): Mirror 1: 0x47615320 Mirror 4: 0x47615320 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee72: skipped 000164 1779127070.831776 punch from 0x02ff38 to 0x03cc50 (0xcd19 bytes) 000165 1779127070.931327 mapread 0x016e39 thru 0x01fff4 (0x91bc bytes) 000167 1779127071.001510 mapwrite 0x035b89 thru 0x03ffff (0xa477 bytes) 000168 1779127071.083743 punch from 0x03323e to 0x03a8e5 (0x76a8 bytes) 000169 1779127071.091591 read 0x02c24f thru 0x02f41a (0x31cc bytes) 000170 1779127071.097365 write 0x039216 thru 0x03ba70 (0x285b bytes) 000171 1779127071.115089 write 0x01ba32 thru 0x029919 (0xdee8 bytes) 000172 1779127071.129428 fallocate 0x00c31f thru 0x00f618 (0x32fa bytes) 000173 1779127071.134128 trunc from 0x02dce0 to 0x03c547 (0xe868 bytes) 000174 1779127071.147368 trunc from 0x040000 to 0x01c579 (0x23a86 bytes) 000175 1779127071.165977 read 0x00505f thru 0x014ca9 (0xfc4b bytes) 000176 1779127071.174811 fallocate 0x004d31 thru 0x008b2f (0x3dff bytes) 000177 1779127071.182378 read 0x01b96d thru 0x01c579 (0xc0d bytes) 000178 1779127071.199728 MIRROR_RESYNC 4 mirrors 000180 1779127072.374014 trunc from 0x01c57a to 0x02fae8 (0x1356f bytes) 000181 1779127072.545721 punch from 0x010bb5 to 0x01c95e (0xbdaa bytes) 000182 1779127072.559881 mapread 0x02cc26 thru 0x02fae8 (0x2ec3 bytes) 000183 1779127072.625634 mapread 0x0172c3 thru 0x018be4 (0x1922 bytes) 000184 1779127072.676583 punch from 0x020aa7 to 0x02f775 (0xeccf bytes) 000185 1779127072.689881 mapwrite 0x02275a thru 0x0297f6 (0x709d bytes) 000186 1779127072.750621 fallocate 0x020706 thru 0x02ef12 (0xe80d bytes) 000187 1779127072.757588 mapwrite 0x025092 thru 0x028b5c (0x3acb bytes) 000188 1779127072.805587 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fae9): Mirror 2: 0xab6f89d8 (stale) Mirror 3: 0xab6f89d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fae9): Mirror 2: 0xab6f89d8 (stale) Mirror 3: 0xab6f89d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fae9: skipped 000189 1779127073.465530 write 0x0270af thru 0x02cfd6 (0x5f28 bytes) 000190 1779127073.504556 fallocate 0x01f1c6 thru 0x022958 (0x3793 bytes) 000191 1779127073.521780 trunc from 0x01dcd3 to 0x025a42 (0x7d70 bytes) 000192 1779127073.531184 fallocate 0x018d70 thru 0x025bfd (0xce8e bytes) 000193 1779127073.556608 mapread 0x02eced thru 0x02fae8 (0xdfc bytes) 000194 1779127073.573255 mapread 0x02bcd3 thru 0x02c26f (0x59d bytes) 000195 1779127073.574202 punch from 0x014094 to 0x023df0 (0xfd5d bytes) 000196 1779127073.589442 punch from 0x00effc to 0x00f3c0 (0x3c5 bytes) 000198 1779127073.598748 write 0x033b47 thru 0x03ffff (0xc4b9 bytes) 000199 1779127073.601682 write 0x020cd8 thru 0x02c794 (0xbabd bytes) 000200 1779127073.628496 trunc from 0x040000 to 0x03c902 (0x36fd bytes) 000201 1779127073.646987 mapwrite 0x01e170 thru 0x02b415 (0xd2a6 bytes) 000202 1779127073.673249 trunc from 0x01c3e8 to 0x022899 (0x64b2 bytes) 000203 1779127073.678740 mapread 0x021422 thru 0x026d1f (0x58fe bytes) 000204 1779127073.708180 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c903): Mirror 2: 0x729685dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c903): Mirror 2: 0x729685dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c903: skipped 000205 1779127074.122339 punch from 0x03ad41 to 0x03c902 (0x1bc2 bytes) 000206 1779127074.132409 mapread 0x00142b thru 0x0024fe (0x10d4 bytes) 000207 1779127074.159668 trunc from 0x01628d to 0x016dde (0xb52 bytes) 000208 1779127074.179117 MIRROR_RESYNC 2 mirrors 000210 1779127075.327854 read 0x037702 thru 0x03c902 (0x5201 bytes) 000211 1779127075.344529 trunc from 0x01f017 to 0x023934 (0x491e bytes) 000212 1779127075.406728 trunc from 0x02be0b to 0x02fd05 (0x3efb bytes) 000213 1779127075.424936 read 0x036147 thru 0x03c902 (0x67bc bytes) 000214 1779127075.452055 mapread 0x00ec99 thru 0x01b380 (0xc6e8 bytes) 000215 1779127075.454200 trunc from 0x03c903 to 0x03ccec (0x3ea bytes) 000216 1779127075.481440 punch from 0x00e385 to 0x01b18e (0xce0a bytes) 000217 1779127075.487859 fallocate 0x011786 thru 0x01e063 (0xc8de bytes) 000218 1779127075.500433 fallocate 0x000d07 thru 0x00ab81 (0x9e7b bytes) 000220 1779127075.507562 mapread 0x013365 thru 0x01461e (0x12ba bytes) 000221 1779127075.518005 mapwrite 0x02342a thru 0x02891f (0x54f6 bytes) 000222 1779127075.544530 read 0x02b675 thru 0x036cbe (0xb64a bytes) 000223 1779127075.545381 fallocate 0x021572 thru 0x02828c (0x6d1b bytes) 000224 1779127075.565435 mapread 0x01ee39 thru 0x0219fc (0x2bc4 bytes) 000225 1779127075.566572 write 0x0072d4 thru 0x01302d (0xbd5a bytes) 000226 1779127075.568688 trunc from 0x03cced to 0x010090 (0x2cc5c bytes) 000227 1779127075.591101 fallocate 0x01b91d thru 0x02ad05 (0xf3e9 bytes) 000228 1779127075.603296 MIRROR_RESYNC 2 mirrors 000229 1779127076.159564 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000230 1779127076.323156 fallocate 0x0028f7 thru 0x005b63 (0x326d bytes) 000231 1779127076.353163 trunc from 0x01e217 to 0x025876 (0x7660 bytes) 000232 1779127076.362629 trunc from 0x02ad06 to 0x02d38a (0x2685 bytes) 000233 1779127076.445701 mapwrite 0x034045 thru 0x03ffff (0xbfbb bytes) 000234 1779127076.658568 write 0x011d5f thru 0x0145e1 (0x2883 bytes) 000235 1779127076.674080 read 0x0284ba thru 0x0314d8 (0x901f bytes) 000239 1779127076.703945 trunc from 0x040000 to 0x0240cd (0x1bf32 bytes) 000240 1779127076.742955 mapread 0x0205a4 thru 0x0240cd (0x3b2a bytes) 000241 1779127076.787419 MIRROR_EXTEND to 2 mirrors 000242 1779127077.460327 write 0x030761 thru 0x03dd38 (0xd5d8 bytes) 000243 1779127077.523950 mapwrite 0x03cfd6 thru 0x03ffff (0x302a bytes) 000244 1779127077.639245 trunc from 0x040000 to 0x02874a (0x178b5 bytes) 000245 1779127077.659546 trunc from 0x00c114 to 0x0151eb (0x90d8 bytes) 000246 1779127077.670467 mapwrite 0x036d94 thru 0x037580 (0x7ed bytes) 000248 1779127077.714303 trunc from 0x028195 to 0x02c1eb (0x4057 bytes) 000250 1779127077.723774 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000252 1779127077.853373 write 0x0389fe thru 0x03ffff (0x7602 bytes) 000253 1779127077.862559 mapwrite 0x0097c9 thru 0x00a47b (0xcb3 bytes) 000254 1779127077.898077 fallocate 0x007242 thru 0x00cc2a (0x59e9 bytes) 000255 1779127077.902336 mapwrite 0x03c2b0 thru 0x03ed71 (0x2ac2 bytes) 000256 1779127077.917110 MIRROR_EXTEND to 2 mirrors 000257 1779127078.705834 trunc from 0x040000 to 0x000b07 (0x3f4f8 bytes) 000258 1779127078.780749 punch from 0x000342 to 0x000b07 (0x7c6 bytes) 000259 1779127078.796160 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000260 1779127079.008424 write 0x023ee6 thru 0x03224a (0xe365 bytes) 000261 1779127079.018149 punch from 0x0286fe to 0x03224a (0x9b4d bytes) 000263 1779127079.051222 write 0x02dce3 thru 0x03b45e (0xd77c bytes) 000264 1779127079.059302 trunc from 0x02c0de to 0x03b45e (0xf381 bytes) 000265 1779127079.089221 punch from 0x0190f8 to 0x025be7 (0xcaf0 bytes) 000266 1779127079.094673 read 0x038832 thru 0x03a05d (0x182c bytes) 000267 1779127079.115509 read 0x0013f3 thru 0x00782d (0x643b bytes) 000268 1779127079.127388 trunc from 0x009ab6 to 0x01614d (0xc698 bytes) 000269 1779127079.132361 mapwrite 0x03ebc3 thru 0x03ffff (0x143d bytes) 000270 1779127079.171057 mapwrite 0x03f0e4 thru 0x03ffff (0xf1c bytes) 000271 1779127079.184405 fallocate 0x034cd4 thru 0x03ffff (0xb32c bytes) 000272 1779127079.197954 fallocate 0x017dd9 thru 0x01a1cc (0x23f4 bytes) 000273 1779127079.212769 trunc from 0x040000 to 0x01c26f (0x23d90 bytes) 000275 1779127079.241505 trunc from 0x00072d to 0x00e3bf (0xdc93 bytes) 000276 1779127079.250450 mapwrite 0x02a6fd thru 0x031d8f (0x7693 bytes) 000277 1779127079.404566 trunc from 0x031d90 to 0x01cbc2 (0x151cd bytes) 000278 1779127079.435173 mapwrite 0x0359d4 thru 0x0375e4 (0x1c11 bytes) 000279 1779127079.516493 trunc from 0x0375e5 to 0x001223 (0x363c1 bytes) 000280 1779127079.547119 mapread 0x000476 thru 0x001223 (0xdae bytes) 000281 1779127079.564441 punch from 0x00068f to 0x001223 (0xb95 bytes) 000282 1779127079.569706 trunc from 0x001224 to 0x030b14 (0x2f8f1 bytes) 000283 1779127079.586034 mapread 0x00f5ec thru 0x01bd93 (0xc7a8 bytes) 000284 1779127079.690589 MIRROR_EXTEND to 2 mirrors 000285 1779127080.568393 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000286 1779127080.779784 mapwrite 0x015458 thru 0x023fae (0xeb57 bytes) 000287 1779127080.854722 write 0x0057e8 thru 0x00dad3 (0x82ec bytes) 000288 1779127080.883588 fallocate 0x0147f3 thru 0x015800 (0x100e bytes) 000289 1779127080.891617 write 0x00d307 thru 0x014504 (0x71fe bytes) 000290 1779127080.897324 read 0x00e9b9 thru 0x019329 (0xa971 bytes) 000291 1779127080.899796 write 0x03fe34 thru 0x03ffff (0x1cc bytes) 000292 1779127080.902418 mapread 0x01154c thru 0x019e28 (0x88dd bytes) 000293 1779127080.904809 write 0x03086e thru 0x030efe (0x691 bytes) 000294 1779127080.914121 read 0x009961 thru 0x01633a (0xc9da bytes) 000296 1779127080.916788 trunc from 0x002d5b to 0x00f9e7 (0xcc8d bytes) 000297 1779127080.943765 trunc from 0x01fac7 to 0x02a231 (0xa76b bytes) 000298 1779127080.957691 write 0x01c0a2 thru 0x0249a5 (0x8904 bytes) 000299 1779127080.971756 read 0x026bf8 thru 0x034213 (0xd61c bytes) 000300 1779127080.985939 read 0x036293 thru 0x03cdb3 (0x6b21 bytes) 000302 1779127080.987738 fallocate 0x022144 thru 0x02fbb5 (0xda72 bytes) 000303 1779127080.994416 fallocate 0x01ecde thru 0x026010 (0x7333 bytes) 000304 1779127081.005074 trunc from 0x040000 to 0x01019c (0x2fe63 bytes) 000305 1779127081.023638 read 0x00b14c thru 0x01019c (0x5051 bytes) 000306 1779127081.024616 fallocate 0x02f96a thru 0x03656d (0x6c04 bytes) 000307 1779127081.030652 mapread 0x006a93 thru 0x01019c (0x970a bytes) 000308 1779127081.031884 mapwrite 0x02c386 thru 0x03bce3 (0xf95e bytes) 000309 1779127081.085121 trunc from 0x03bce4 to 0x00059a (0x3b749 bytes) 000310 1779127081.112354 fallocate 0x037314 thru 0x03f7f7 (0x84e4 bytes) 000311 1779127081.120527 write 0x029c13 thru 0x0313f5 (0x77e3 bytes) 000313 1779127081.145387 write 0x030af1 thru 0x03c7d9 (0xbce9 bytes) 000314 1779127081.146962 mapwrite 0x0241cd thru 0x033ae4 (0xf918 bytes) 000315 1779127081.224749 mapwrite 0x01a6da thru 0x01fb34 (0x545b bytes) 000316 1779127081.298808 MIRROR_EXTEND to 2 mirrors 000317 1779127082.028645 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000318 1779127082.163170 read 0x00cb87 thru 0x00de6d (0x12e7 bytes) 000319 1779127082.169790 mapread 0x03be42 thru 0x03f7f7 (0x39b6 bytes) 000320 1779127082.193769 mapread 0x01a580 thru 0x01f851 (0x52d2 bytes) 000321 1779127082.204608 trunc from 0x03f335 to 0x03f7f7 (0x4c3 bytes) 000322 1779127082.224558 read 0x028d3b thru 0x02f910 (0x6bd6 bytes) 000323 1779127082.257952 write 0x00ff5b thru 0x015f4e (0x5ff4 bytes) 000324 1779127082.264166 trunc from 0x017a00 to 0x01d80a (0x5e0b bytes) 000325 1779127082.271576 read 0x001566 thru 0x011554 (0xffef bytes) 000326 1779127082.272801 trunc from 0x037ad2 to 0x03f7f7 (0x7d26 bytes) 000327 1779127082.279128 read 0x0286db thru 0x02b373 (0x2c99 bytes) 000328 1779127082.279954 fallocate 0x004635 thru 0x00979a (0x5166 bytes) 000329 1779127082.285372 read 0x010f77 thru 0x01fed3 (0xef5d bytes) 000330 1779127082.295633 trunc from 0x03f7f8 to 0x0137dd (0x2c01a bytes) 000331 1779127082.335855 mapwrite 0x0098a8 thru 0x0138b1 (0xa00a bytes) 000332 1779127082.372441 fallocate 0x00079c thru 0x0020c6 (0x192b bytes) 000333 1779127082.378349 mapread 0x005125 thru 0x0069ad (0x1889 bytes) 000334 1779127082.380395 fallocate 0x030f1c thru 0x03fa51 (0xeb36 bytes) 000335 1779127082.386055 trunc from 0x0334d3 to 0x039d42 (0x6870 bytes) 000336 1779127082.391356 punch from 0x02e8c0 to 0x03d61b (0xed5c bytes) 000337 1779127082.396974 read 0x00afff thru 0x0163c1 (0xb3c3 bytes) 000338 1779127082.413686 MIRROR_EXTEND to 2 mirrors 000339 1779127083.156602 mapread 0x02b951 thru 0x0303c6 (0x4a76 bytes) 000340 1779127083.186302 trunc from 0x03fa52 to 0x02f21c (0x10835 bytes) 000342 1779127083.295777 punch from 0x02b275 to 0x02c629 (0x13b5 bytes) 000343 1779127083.332200 write 0x02fcd5 thru 0x03b7fa (0xbb26 bytes) 000344 1779127083.362666 mapread 0x037fbe thru 0x03a5eb (0x262e bytes) 000345 1779127083.365100 trunc from 0x02cab5 to 0x03b7fa (0xed46 bytes) 000346 1779127083.409757 write 0x038960 thru 0x03ffff (0x76a0 bytes) 000347 1779127083.433462 punch from 0x03ca88 to 0x03e44d (0x19c6 bytes) 000348 1779127083.459817 write 0x03cd2c thru 0x03ffff (0x32d4 bytes) 000349 1779127083.470804 read 0x014de0 thru 0x01a337 (0x5558 bytes) 000350 1779127083.481555 mapread 0x01ccbe thru 0x02153c (0x487f bytes) 000351 1779127083.506744 mapwrite 0x0136e2 thru 0x014a45 (0x1364 bytes) 000352 1779127083.541190 trunc from 0x02533d to 0x02544d (0x111 bytes) 000353 1779127083.553733 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000354 1779127083.749832 fallocate 0x035443 thru 0x03ffff (0xabbd bytes) 000355 1779127083.758327 punch from 0x0096b6 to 0x00ba8c (0x23d7 bytes) 000356 1779127083.769531 write 0x023a81 thru 0x0279d2 (0x3f52 bytes) 000358 1779127083.795717 trunc from 0x040000 to 0x037d19 (0x82e6 bytes) 000359 1779127083.811685 MIRROR_EXTEND to 2 mirrors 000360 1779127084.824528 fallocate 0x01b055 thru 0x024016 (0x8fc2 bytes) 000361 1779127084.889684 punch from 0x028550 to 0x03273c (0xa1ed bytes) 000362 1779127084.897654 mapread 0x02ea5b thru 0x03787d (0x8e23 bytes) 000363 1779127084.916805 read 0x00126f thru 0x00ebd3 (0xd965 bytes) 000364 1779127084.929969 fallocate 0x02c74c thru 0x03139d (0x4c52 bytes) 000365 1779127084.935712 trunc from 0x037d1a to 0x02dba3 (0xa176 bytes) 000366 1779127084.959693 mapread 0x01e66f thru 0x022dd9 (0x476b bytes) 000367 1779127084.968403 mapwrite 0x022cd8 thru 0x028242 (0x556b bytes) 000368 1779127084.993933 trunc from 0x02dba4 to 0x01042b (0x1d778 bytes) 000369 1779127085.018224 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000370 1779127085.300082 fallocate 0x0305d2 thru 0x039dca (0x97f9 bytes) 000371 1779127085.310507 trunc from 0x039dcb to 0x0074fb (0x328cf bytes) 000372 1779127085.358695 read 0x004e28 thru 0x0074fb (0x26d4 bytes) 000373 1779127085.372663 write 0x012872 thru 0x018259 (0x59e8 bytes) 000374 1779127085.377851 trunc from 0x004810 to 0x00df8c (0x977d bytes) 000376 1779127085.385068 fallocate 0x003e51 thru 0x01019c (0xc34c bytes) 000377 1779127085.391864 fallocate 0x020447 thru 0x02d396 (0xcf50 bytes) 000378 1779127085.401336 write 0x02c2a4 thru 0x03207a (0x5dd7 bytes) 000379 1779127085.404385 trunc from 0x000393 to 0x00c71d (0xc38b bytes) 000383 1779127085.422210 mapwrite 0x01dff1 thru 0x01f42c (0x143c bytes) 000384 1779127085.482548 mapwrite 0x023abb thru 0x026f4d (0x3493 bytes) 000385 1779127085.513665 trunc from 0x022310 to 0x02714a (0x4e3b bytes) 000386 1779127085.545733 read 0x014611 thru 0x017e9c (0x388c bytes) 000389 1779127085.549192 read 0x028b77 thru 0x03207a (0x9504 bytes) 000390 1779127085.577120 trunc from 0x00f80d to 0x01ee2c (0xf620 bytes) 000391 1779127085.587684 mapread 0x0089ed thru 0x013145 (0xa759 bytes) 000392 1779127085.671154 MIRROR_EXTEND to 2 mirrors 000393 1779127086.498492 mapwrite 0x012122 thru 0x01b7b9 (0x9698 bytes) 000394 1779127086.632467 mapread 0x008c2f thru 0x00f4aa (0x687c bytes) 000395 1779127086.651475 read 0x00d6a3 thru 0x01085f (0x31bd bytes) 000397 1779127086.655515 fallocate 0x02e82f thru 0x030ee5 (0x26b7 bytes) 000398 1779127086.666805 punch from 0x02f04a to 0x03207a (0x3031 bytes) 000399 1779127086.682789 trunc from 0x03207b to 0x001d5a (0x30320 bytes) 000400 1779127086.708010 mapread 0x0006f8 thru 0x001d5a (0x1663 bytes) 000401 1779127086.713708 trunc from 0x001d5b to 0x02f9df (0x2dc85 bytes) 000402 1779127086.728330 write 0x0010f2 thru 0x00f7c9 (0xe6d8 bytes) 000404 1779127086.736552 fallocate 0x023ff9 thru 0x02f48e (0xb496 bytes) 000405 1779127086.741379 trunc from 0x017a4e to 0x025b01 (0xe0b4 bytes) 000406 1779127086.754210 write 0x036ad0 thru 0x03ffff (0x9530 bytes) 000407 1779127086.770333 MIRROR_RESYNC 2 mirrors 000408 1779127087.578572 punch from 0x01271b to 0x01cfd6 (0xa8bc bytes) 000409 1779127087.625403 punch from 0x00bb4a to 0x00f731 (0x3be8 bytes) 000410 1779127087.635220 mapread 0x01b55c thru 0x020837 (0x52dc bytes) 000411 1779127087.663172 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000412 1779127087.944047 punch from 0x010846 to 0x011c0d (0x13c8 bytes) 000413 1779127087.950599 trunc from 0x02044c to 0x0271ec (0x6da1 bytes) 000414 1779127087.958067 trunc from 0x040000 to 0x023136 (0x1cec9 bytes) 000415 1779127087.985670 punch from 0x000f46 to 0x006d06 (0x5dc1 bytes) 000416 1779127087.992120 mapread 0x020319 thru 0x020909 (0x5f1 bytes) 000417 1779127087.999075 write 0x002d2f thru 0x00d3bf (0xa691 bytes) 000418 1779127088.011474 mapread 0x00a93d thru 0x015325 (0xa9e9 bytes) 000419 1779127088.022432 write 0x039124 thru 0x03ffff (0x6edc bytes) 000421 1779127088.025517 trunc from 0x040000 to 0x01718b (0x28e74 bytes) 000422 1779127088.040788 punch from 0x005201 to 0x00f296 (0xa096 bytes) 000423 1779127088.060138 trunc from 0x00d09e to 0x01718b (0xa0ee bytes) 000424 1779127088.070560 trunc from 0x01155a to 0x01718b (0x5c32 bytes) 000426 1779127088.081614 mapwrite 0x03e1ff thru 0x03ffff (0x1e01 bytes) 000427 1779127088.150638 trunc from 0x040000 to 0x03b707 (0x48f8 bytes) 000428 1779127088.182329 mapread 0x02357b thru 0x026697 (0x311d bytes) 000429 1779127088.194804 punch from 0x02282c to 0x030485 (0xdc5a bytes) 000430 1779127088.200837 mapwrite 0x0082e3 thru 0x01396a (0xb688 bytes) 000431 1779127088.250217 trunc from 0x03b708 to 0x00a392 (0x31375 bytes) 000432 1779127088.281539 read 0x000692 thru 0x00a392 (0x9d01 bytes) 000434 1779127088.288184 trunc from 0x006107 to 0x00a392 (0x428c bytes) 000435 1779127088.297286 trunc from 0x00a393 to 0x02b56e (0x211dc bytes) 000436 1779127088.313013 trunc from 0x02b56f to 0x01fb0f (0xba5f bytes) 000437 1779127088.332091 trunc from 0x008171 to 0x00f6b1 (0x7541 bytes) 000438 1779127088.338691 trunc from 0x01bdbd to 0x01fb0f (0x3d53 bytes) 000439 1779127088.344125 mapwrite 0x0378cb thru 0x03ffff (0x8735 bytes) 000440 1779127088.402454 write 0x00a100 thru 0x00c8d4 (0x27d5 bytes) 000444 1779127088.422270 mapwrite 0x039e32 thru 0x03a6f4 (0x8c3 bytes) 000445 1779127088.441484 trunc from 0x019d23 to 0x0266c7 (0xc9a5 bytes) 000446 1779127088.447229 mapread 0x00d921 thru 0x01168e (0x3d6e bytes) 000450 1779127088.456404 fallocate 0x01518c thru 0x020331 (0xb1a6 bytes) 000451 1779127088.462788 fallocate 0x02a9fd thru 0x035aaa (0xb0ae bytes) 000453 1779127088.471685 trunc from 0x040000 to 0x004e30 (0x3b1cf bytes) 000454 1779127088.496843 mapwrite 0x002059 thru 0x003f04 (0x1eac bytes) 000455 1779127088.512311 punch from 0x001914 to 0x004e30 (0x351d bytes) 000456 1779127088.518796 trunc from 0x004e31 to 0x01aabc (0x15c8c bytes) 000457 1779127088.539031 punch from 0x010398 to 0x0177de (0x7447 bytes) 000460 1779127088.547354 trunc from 0x00ec1f to 0x01aabc (0xbe9e bytes) 000461 1779127088.553141 punch from 0x012124 to 0x018662 (0x653f bytes) 000463 1779127088.562554 trunc from 0x0109d0 to 0x01aabc (0xa0ed bytes) 000464 1779127088.568398 trunc from 0x016728 to 0x017e01 (0x16da bytes) 000465 1779127088.573520 punch from 0x015ef3 to 0x01aabc (0x4bca bytes) 000466 1779127088.584708 write 0x03a60d thru 0x03ffff (0x59f3 bytes) 000468 1779127088.589422 mapread 0x02377f thru 0x026287 (0x2b09 bytes) 000469 1779127088.600314 trunc from 0x040000 to 0x03124c (0xedb3 bytes) 000470 1779127088.619686 mapwrite 0x008af5 thru 0x018326 (0xf832 bytes) 000471 1779127088.663065 mapread 0x013969 thru 0x015769 (0x1e01 bytes) 000472 1779127088.665498 fallocate 0x0347c6 thru 0x038db2 (0x45ed bytes) 000473 1779127088.671409 trunc from 0x03124d to 0x02017d (0x110cf bytes) 000474 1779127088.690723 trunc from 0x02017e to 0x0025ae (0x1dbcf bytes) 000475 1779127088.714013 trunc from 0x0025af to 0x01462a (0x1207c bytes) 000477 1779127088.738778 write 0x035ea3 thru 0x03ffff (0xa15d bytes) 000478 1779127088.741780 punch from 0x00ac9a to 0x010378 (0x56df bytes) 000479 1779127088.747727 trunc from 0x026552 to 0x0315d8 (0xb087 bytes) 000481 1779127088.755066 mapwrite 0x0183e3 thru 0x018ce5 (0x903 bytes) 000482 1779127088.786363 mapread 0x0046bf thru 0x00f9cb (0xb30d bytes) 000484 1779127088.804551 trunc from 0x040000 to 0x01a9e8 (0x25617 bytes) 000485 1779127088.830510 trunc from 0x0193f3 to 0x01a9e8 (0x15f6 bytes) 000486 1779127088.840147 fallocate 0x028e89 thru 0x036485 (0xd5fd bytes) 000488 1779127088.849541 mapread 0x033f30 thru 0x036485 (0x2556 bytes) 000489 1779127088.866746 write 0x000bad thru 0x00e9c5 (0xde19 bytes) 000490 1779127088.869610 write 0x00a4ac thru 0x00b83b (0x1390 bytes) 000491 1779127088.872539 mapwrite 0x02fe9f thru 0x03d2e4 (0xd446 bytes) 000492 1779127088.928586 trunc from 0x03d2e5 to 0x02fff1 (0xd2f3 bytes) 000493 1779127088.951284 punch from 0x020d26 to 0x024e5d (0x4138 bytes) 000494 1779127088.959950 trunc from 0x02fff2 to 0x01588e (0x1a763 bytes) 000496 1779127089.021312 mapread 0x012400 thru 0x01588e (0x348f bytes) 000497 1779127089.034501 MIRROR_EXTEND to 2 mirrors 000498 1779127089.934005 punch from 0x014314 to 0x01588e (0x157b bytes) 000499 1779127090.001385 trunc from 0x01179f to 0x01588e (0x40f0 bytes) 000500 1779127090.011436 write 0x017a01 thru 0x01ec83 (0x7283 bytes) 000501 1779127090.020513 write 0x0268e7 thru 0x02bcbd (0x53d7 bytes) 000502 1779127090.034864 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2bcbe): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bcbe): Mirror 2: 0x9d192a20 (stale) CRC-32 checksum value for chunk [0, 0x2bcbe): Mirror 1: 0x5df293db Mirror 3: 0x5df293db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bcbe: skipped 000503 1779127091.419664 mapwrite 0x01436c thru 0x0144eb (0x180 bytes) 000504 1779127091.569630 trunc from 0x02bcbe to 0x03b7a2 (0xfae5 bytes) 000505 1779127091.610066 write 0x03e9b5 thru 0x03ffff (0x164b bytes) 000506 1779127091.611202 fallocate 0x008492 thru 0x0132b0 (0xae1f bytes) 000507 1779127091.622096 punch from 0x0152d1 to 0x019d60 (0x4a90 bytes) 000508 1779127091.632881 punch from 0x011b35 to 0x01a5be (0x8a8a bytes) 000509 1779127091.644836 trunc from 0x032994 to 0x038d79 (0x63e6 bytes) 000510 1779127091.652029 punch from 0x029878 to 0x037301 (0xda8a bytes) 000511 1779127091.678391 mapread 0x002000 thru 0x00e915 (0xc916 bytes) 000513 1779127091.696118 fallocate 0x000a1f thru 0x00f798 (0xed7a bytes) 000514 1779127091.719877 trunc from 0x00ae46 to 0x01619d (0xb358 bytes) 000515 1779127091.731085 write 0x0139c4 thru 0x0159d8 (0x2015 bytes) 000516 1779127091.754993 trunc from 0x040000 to 0x03ccb2 (0x334d bytes) 000517 1779127091.782034 trunc from 0x02e2cb to 0x034d79 (0x6aaf bytes) 000518 1779127091.795010 trunc from 0x0250ad to 0x02a99b (0x58ef bytes) 000519 1779127091.809506 punch from 0x0159e6 to 0x019b2c (0x4147 bytes) 000520 1779127091.823268 MIRROR_RESYNC 3 mirrors 000521 1779127093.483336 fallocate 0x02057a thru 0x0241a1 (0x3c28 bytes) 000522 1779127093.528620 punch from 0x006140 to 0x00c2e6 (0x61a7 bytes) 000523 1779127093.536503 write 0x003f39 thru 0x00fc50 (0xbd18 bytes) 000524 1779127093.549295 punch from 0x0296d4 to 0x03535a (0xbc87 bytes) 000525 1779127093.556624 read 0x00ef51 thru 0x0187c9 (0x9879 bytes) 000526 1779127093.568582 trunc from 0x000566 to 0x00575e (0x51f9 bytes) 000527 1779127093.598515 trunc from 0x0208d0 to 0x02f578 (0xeca9 bytes) 000528 1779127093.605752 write 0x03852e thru 0x03f865 (0x7338 bytes) 000529 1779127093.626658 fallocate 0x030b07 thru 0x03fca8 (0xf1a2 bytes) 000530 1779127093.636671 trunc from 0x03f866 to 0x0282f3 (0x17572 bytes) 000531 1779127093.689744 fallocate 0x011106 thru 0x01eca1 (0xdb9c bytes) 000532 1779127093.708775 mapread 0x00c076 thru 0x0132e3 (0x726e bytes) 000533 1779127093.711550 write 0x004029 thru 0x00de1d (0x9df5 bytes) 000534 1779127093.740114 write 0x002469 thru 0x010a5e (0xe5f6 bytes) 000535 1779127093.745065 mapread 0x0111ef thru 0x016b04 (0x5916 bytes) 000536 1779127093.750652 trunc from 0x0282f4 to 0x02128d (0x7066 bytes) 000537 1779127093.783955 write 0x0334b7 thru 0x037a36 (0x4580 bytes) 000538 1779127093.785443 read 0x0329dd thru 0x037a36 (0x505a bytes) 000539 1779127093.828613 mapwrite 0x024deb thru 0x033e05 (0xf01b bytes) 000540 1779127093.946711 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37a37): Mirror 3: 0x261ea3fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a37): Mirror 3: 0x261ea3fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a37: skipped 000541 1779127094.466118 fallocate 0x0253f8 thru 0x02d9be (0x85c7 bytes) 000542 1779127094.476989 write 0x01de92 thru 0x024701 (0x6870 bytes) 000543 1779127094.497826 mapwrite 0x0285a8 thru 0x02aeee (0x2947 bytes) 000544 1779127094.542026 mapread 0x0370ee thru 0x037a36 (0x949 bytes) 000545 1779127094.551646 write 0x01c33c thru 0x029b16 (0xd7db bytes) 000546 1779127094.565619 punch from 0x0346df to 0x037a36 (0x3358 bytes) 000547 1779127094.572671 write 0x01aa80 thru 0x01ca72 (0x1ff3 bytes) 000548 1779127094.574716 read 0x02d6dc thru 0x02dca9 (0x5ce bytes) 000549 1779127094.577359 punch from 0x027679 to 0x034df1 (0xd779 bytes) 000550 1779127094.603173 fallocate 0x005f9a thru 0x007328 (0x138f bytes) 000551 1779127094.610701 read 0x00bd69 thru 0x00d636 (0x18ce bytes) 000552 1779127094.624391 read 0x021149 thru 0x024131 (0x2fe9 bytes) 000553 1779127094.630017 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000554 1779127094.793076 write 0x01e964 thru 0x021751 (0x2dee bytes) 000555 1779127094.808668 write 0x02a720 thru 0x02f371 (0x4c52 bytes) 000556 1779127094.822165 mapwrite 0x00485a thru 0x00994b (0x50f2 bytes) 000557 1779127094.880561 trunc from 0x037a37 to 0x00d27f (0x2a7b7 bytes) 000558 1779127094.913523 fallocate 0x023064 thru 0x0259a7 (0x2944 bytes) 000560 1779127094.921171 write 0x01bf66 thru 0x020348 (0x43e3 bytes) 000561 1779127094.925330 mapread 0x01a377 thru 0x01f3a1 (0x502b bytes) 000562 1779127094.945117 write 0x01707d thru 0x01ff82 (0x8f06 bytes) 000563 1779127094.947794 mapread 0x0040c7 thru 0x00fd24 (0xbc5e bytes) 000564 1779127094.959420 punch from 0x012c39 to 0x020348 (0xd710 bytes) 000565 1779127094.981588 trunc from 0x014f12 to 0x016e60 (0x1f4f bytes) 000566 1779127094.988594 fallocate 0x001459 thru 0x0057c0 (0x4368 bytes) 000567 1779127094.996992 fallocate 0x004227 thru 0x007aba (0x3894 bytes) 000568 1779127095.007126 write 0x039528 thru 0x03ffff (0x6ad8 bytes) 000569 1779127095.011026 mapread 0x01248b thru 0x01f0f3 (0xcc69 bytes) 000570 1779127095.041682 trunc from 0x040000 to 0x03429d (0xbd62 bytes) 000571 1779127095.064614 trunc from 0x008a97 to 0x0181e1 (0xf74b bytes) 000572 1779127095.075720 mapread 0x02ad9e thru 0x031033 (0x6296 bytes) 000574 1779127095.109744 punch from 0x0262bb to 0x032b22 (0xc868 bytes) 000575 1779127095.118476 write 0x036acb thru 0x03e069 (0x759f bytes) 000577 1779127095.127424 mapread 0x02c53a thru 0x03969a (0xd161 bytes) 000578 1779127095.152332 read 0x002459 thru 0x00c611 (0xa1b9 bytes) 000579 1779127095.178369 read 0x00b186 thru 0x015bff (0xaa7a bytes) 000581 1779127095.205723 trunc from 0x022e5f to 0x02cb68 (0x9d0a bytes) 000582 1779127095.242410 MIRROR_EXTEND to 2 mirrors 000583 1779127096.481578 trunc from 0x00581d to 0x012e52 (0xd636 bytes) 000584 1779127096.523533 write 0x032d4c thru 0x03a13d (0x73f2 bytes) 000585 1779127096.551174 trunc from 0x02b831 to 0x02c9e0 (0x11b0 bytes) 000586 1779127096.583739 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e06a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e06a): Mirror 2: 0xce5bf167 (stale) CRC-32 checksum value for chunk [0, 0x3e06a): Mirror 1: 0xa066f2b2 Mirror 3: 0xa066f2b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e06a: skipped 000587 1779127098.055883 trunc from 0x03e06a to 0x02b9a7 (0x126c2 bytes) 000588 1779127098.114566 trunc from 0x01f03b to 0x02b9a7 (0xc96d bytes) 000589 1779127098.119766 punch from 0x022afc to 0x028d74 (0x6279 bytes) 000590 1779127098.125741 mapread 0x01a642 thru 0x01fff7 (0x59b6 bytes) 000591 1779127098.156693 trunc from 0x02b9a8 to 0x03b920 (0xff79 bytes) 000592 1779127098.184233 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b921): Mirror 2: 0x9334825e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b921): Mirror 2: 0x9334825e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b921: skipped 000594 1779127098.583665 read 0x01784a thru 0x021bc1 (0xa378 bytes) 000595 1779127098.594593 punch from 0x0054f1 to 0x011b23 (0xc633 bytes) 000596 1779127098.599379 punch from 0x008b1d to 0x00e002 (0x54e6 bytes) 000597 1779127098.604546 write 0x00fc73 thru 0x016e45 (0x71d3 bytes) 000598 1779127098.616838 fallocate 0x03f99d thru 0x03ffff (0x663 bytes) 000599 1779127098.622213 fallocate 0x00504e thru 0x008fac (0x3f5f bytes) 000600 1779127098.628032 mapwrite 0x03aef7 thru 0x03ffff (0x5109 bytes) 000601 1779127098.669006 trunc from 0x040000 to 0x02c30f (0x13cf0 bytes) 000602 1779127098.723333 mapread 0x016119 thru 0x02292c (0xc814 bytes) 000604 1779127098.773171 write 0x016f48 thru 0x0267c4 (0xf87d bytes) 000605 1779127098.777812 trunc from 0x02c310 to 0x00737e (0x24f91 bytes) 000606 1779127098.830246 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x737f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x737f): Mirror 2: 0xc6ed79f1 (stale) CRC-32 checksum value for chunk [0, 0x737f): Mirror 1: 0x1dbb2186 Mirror 3: 0x1dbb2186 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x737f: skipped 000607 1779127100.086150 write 0x022fd7 thru 0x030929 (0xd953 bytes) 000608 1779127100.149143 trunc from 0x013a86 to 0x01b6f4 (0x7c6f bytes) 000610 1779127100.156495 fallocate 0x0092ec thru 0x017a27 (0xe73c bytes) 000611 1779127100.164444 fallocate 0x01cf43 thru 0x0260fd (0x91bb bytes) 000612 1779127100.174802 trunc from 0x03092a to 0x035abd (0x5194 bytes) 000613 1779127100.213279 mapread 0x01d064 thru 0x02a1cd (0xd16a bytes) 000614 1779127100.242984 MIRROR_RESYNC 3 mirrors 000615 1779127101.460824 mapwrite 0x0220bf thru 0x022db4 (0xcf6 bytes) 000616 1779127101.562495 mapread 0x035383 thru 0x035abd (0x73b bytes) 000617 1779127101.577576 mapread 0x02a042 thru 0x031689 (0x7648 bytes) 000618 1779127101.591529 trunc from 0x0275aa to 0x030310 (0x8d67 bytes) 000619 1779127101.605037 mapwrite 0x009175 thru 0x015623 (0xc4af bytes) 000621 1779127101.673723 mapwrite 0x0290cf thru 0x030b67 (0x7a99 bytes) 000622 1779127101.728624 read 0x019e17 thru 0x0241c8 (0xa3b2 bytes) 000623 1779127101.748862 trunc from 0x035abe to 0x0223c2 (0x136fb bytes) 000624 1779127101.837527 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x223c3): Mirror 3: 0xe4d345ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x223c3): Mirror 3: 0xe4d345ef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x223c3: skipped 000625 1779127102.261821 trunc from 0x01ecb2 to 0x0223c2 (0x3711 bytes) 000626 1779127102.274871 mapread 0x00171f thru 0x0050ee (0x39d0 bytes) 000627 1779127102.314317 trunc from 0x007b57 to 0x00ddf3 (0x629d bytes) 000628 1779127102.323022 punch from 0x0039f5 to 0x004c4a (0x1256 bytes) 000629 1779127102.341686 fallocate 0x021aa0 thru 0x0312a4 (0xf805 bytes) 000630 1779127102.363382 fallocate 0x021311 thru 0x02ab8f (0x987f bytes) 000631 1779127102.385384 MIRROR_RESYNC 2 mirrors 000632 1779127103.213046 write 0x00d280 thru 0x01aa3c (0xd7bd bytes) 000633 1779127103.279669 fallocate 0x02bc93 thru 0x03a7ed (0xeb5b bytes) 000634 1779127103.290407 trunc from 0x0223c3 to 0x01848f (0x9f33 bytes) 000635 1779127103.327494 trunc from 0x018490 to 0x01cb0f (0x4680 bytes) 000636 1779127103.355013 mapwrite 0x0021b6 thru 0x009f04 (0x7d4f bytes) 000637 1779127103.396606 mapwrite 0x03ed5f thru 0x03ffff (0x12a1 bytes) 000638 1779127103.449541 mapread 0x003293 thru 0x00c6ae (0x941c bytes) 000639 1779127103.451965 trunc from 0x0086f9 to 0x012019 (0x9921 bytes) 000640 1779127103.457351 fallocate 0x010736 thru 0x01a708 (0x9fd3 bytes) 000641 1779127103.462604 mapread 0x015ace thru 0x024e1d (0xf350 bytes) 000643 1779127103.488814 write 0x00924a thru 0x00c6cf (0x3486 bytes) 000644 1779127103.510497 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa36da9b0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xca11b06 Mirror 4: 0xca11b06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000645 1779127104.879220 trunc from 0x040000 to 0x01b3a6 (0x24c59 bytes) 000646 1779127104.952684 trunc from 0x011b37 to 0x011d8c (0x256 bytes) 000647 1779127104.973798 write 0x009fbe thru 0x00e459 (0x449c bytes) 000648 1779127104.995706 mapread 0x01a8b8 thru 0x01b3a6 (0xaef bytes) 000649 1779127105.029152 write 0x019849 thru 0x01e51f (0x4cd7 bytes) 000650 1779127105.030447 mapwrite 0x00bdf0 thru 0x00e4ba (0x26cb bytes) 000652 1779127105.079207 mapread 0x003bbe thru 0x00da60 (0x9ea3 bytes) 000653 1779127105.108538 trunc from 0x01e520 to 0x019b48 (0x49d7 bytes) 000655 1779127105.149643 fallocate 0x01bc1a thru 0x01f9dc (0x3dc3 bytes) 000656 1779127105.157446 punch from 0x009f51 to 0x011cbb (0x7d6b bytes) 000657 1779127105.171186 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x19b49): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19b49): Mirror 3: 0xcfe06918 (stale) Mirror 4: 0x9c4232a2 (stale) CRC-32 checksum value for chunk [0, 0x19b49): Mirror 1: 0xffa1b5ec Mirror 5: 0xffa1b5ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19b49: skipped 000658 1779127106.341379 trunc from 0x007226 to 0x016da6 (0xfb81 bytes) 000659 1779127106.406274 trunc from 0x019b49 to 0x02fc43 (0x160fb bytes) 000660 1779127106.432006 trunc from 0x02fc44 to 0x029b8e (0x60b5 bytes) 000661 1779127106.446319 read 0x0229a8 thru 0x029b8e (0x71e7 bytes) 000662 1779127106.455477 mapwrite 0x00f4f5 thru 0x01b210 (0xbd1c bytes) 000663 1779127106.501474 fallocate 0x0250a3 thru 0x02f264 (0xa1c2 bytes) 000664 1779127106.520683 read 0x006c4f thru 0x00b244 (0x45f6 bytes) 000665 1779127106.539974 punch from 0x014fc7 to 0x016155 (0x118f bytes) 000666 1779127106.557633 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x29b8f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b8f): Mirror 3: 0xc1ce89f4 (stale) Mirror 4: 0x192efa76 (stale) Mirror 5: 0xe086727d (stale) CRC-32 checksum value for chunk [0, 0x29b8f): Mirror 1: 0x7633ad09 Mirror 6: 0x7633ad09 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b8f: skipped 000668 1779127108.198991 trunc from 0x0071ca to 0x00dcd1 (0x6b08 bytes) 000669 1779127108.249504 punch from 0x00c61d to 0x00f635 (0x3019 bytes) 000670 1779127108.258140 trunc from 0x00f8a9 to 0x017cc7 (0x841f bytes) 000671 1779127108.265451 trunc from 0x029b8f to 0x0269af (0x31df bytes) 000673 1779127108.320150 read 0x022670 thru 0x0269af (0x4340 bytes) 000674 1779127108.341131 write 0x0353c6 thru 0x03ffff (0xac3a bytes) 000675 1779127108.345284 punch from 0x01ac41 to 0x027e99 (0xd259 bytes) 000676 1779127108.353030 mapwrite 0x0238aa thru 0x02dfbe (0xa715 bytes) 000677 1779127108.420774 fallocate 0x037cac thru 0x039644 (0x1999 bytes) 000678 1779127108.432735 mapread 0x033168 thru 0x03ffff (0xce98 bytes) 000679 1779127108.448482 punch from 0x002180 to 0x008e6c (0x6ced bytes) 000680 1779127108.462525 fallocate 0x03df89 thru 0x03ffff (0x2077 bytes) 000682 1779127108.488587 MIRROR_RESYNC 5 mirrors 000683 1779127110.993648 read 0x01adba thru 0x029103 (0xe34a bytes) 000684 1779127111.014293 MIRROR_RESYNC 5 mirrors 000685 1779127111.586521 read 0x006097 thru 0x015405 (0xf36f bytes) 000686 1779127111.595252 write 0x03ba97 thru 0x03ffff (0x4569 bytes) 000687 1779127111.652047 trunc from 0x020de8 to 0x020f37 (0x150 bytes) 000688 1779127111.666825 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6cbfe5c9 (stale) Mirror 4: 0x6cbfe5c9 (stale) Mirror 6: 0x6cbfe5c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6cbfe5c9 (stale) Mirror 4: 0x6cbfe5c9 (stale) Mirror 6: 0x6cbfe5c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000689 1779127112.215002 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6cbfe5c9 (stale) Mirror 4: 0x6cbfe5c9 (stale) Mirror 6: 0x6cbfe5c9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfc329ba1 Mirror 7: 0xfc329ba1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000690 1779127113.540621 write 0x02b2b4 thru 0x02f823 (0x4570 bytes) 000691 1779127113.600733 trunc from 0x03da2c to 0x03ffff (0x25d4 bytes) 000692 1779127113.616305 punch from 0x02bd3a to 0x031e17 (0x60de bytes) 000693 1779127113.642006 fallocate 0x02738e thru 0x02ce12 (0x5a85 bytes) 000694 1779127113.651943 trunc from 0x040000 to 0x003770 (0x3c88f bytes) 000695 1779127113.673479 punch from 0x0001cc to 0x002db8 (0x2bed bytes) 000696 1779127113.681358 read 0x00329f thru 0x003770 (0x4d2 bytes) 000697 1779127113.688142 mapread 0x000c47 thru 0x003770 (0x2b2a bytes) 000698 1779127113.694638 fallocate 0x0221f1 thru 0x02c76d (0xa57d bytes) 000699 1779127113.701957 mapwrite 0x003405 thru 0x00c8d8 (0x94d4 bytes) 000700 1779127113.745234 mapwrite 0x025e15 thru 0x03541e (0xf60a bytes) 000701 1779127113.853465 mapread 0x02e819 thru 0x03316d (0x4955 bytes) 000702 1779127113.858487 fallocate 0x01cd82 thru 0x02642d (0x96ac bytes) 000703 1779127113.878064 write 0x00c234 thru 0x017656 (0xb423 bytes) 000704 1779127113.909953 MIRROR_RESYNC 5 mirrors 000705 1779127115.260004 read 0x021404 thru 0x028b96 (0x7793 bytes) 000706 1779127115.275093 mapread 0x009cdc thru 0x00d732 (0x3a57 bytes) 000707 1779127115.303023 trunc from 0x03541f to 0x0097d4 (0x2bc4a bytes) 000708 1779127115.352555 fallocate 0x02bc74 thru 0x034bb4 (0x8f41 bytes) 000709 1779127115.372261 mapwrite 0x03aa80 thru 0x03ffff (0x5580 bytes) 000710 1779127115.462736 read 0x001ee7 thru 0x003293 (0x13ad bytes) 000711 1779127115.486451 punch from 0x00113d to 0x007e0e (0x6cd2 bytes) 000712 1779127115.494508 fallocate 0x01abfc thru 0x01f4e9 (0x48ee bytes) 000713 1779127115.501380 mapwrite 0x02349e thru 0x023e1a (0x97d bytes) 000714 1779127115.516781 fallocate 0x027fd9 thru 0x034971 (0xc999 bytes) 000715 1779127115.525414 write 0x020437 thru 0x023f99 (0x3b63 bytes) 000716 1779127115.529574 fallocate 0x00f9ee thru 0x01c049 (0xc65c bytes) 000717 1779127115.541715 read 0x02810f thru 0x0305c2 (0x84b4 bytes) 000718 1779127115.542446 read 0x016dc2 thru 0x01d2b2 (0x64f1 bytes) 000719 1779127115.543114 mapwrite 0x01195d thru 0x013a4c (0x20f0 bytes) 000721 1779127115.595938 read 0x00de1b thru 0x01022f (0x2415 bytes) 000722 1779127115.598237 trunc from 0x040000 to 0x004230 (0x3bdcf bytes) 000723 1779127115.635538 trunc from 0x004231 to 0x02dc84 (0x29a54 bytes) 000724 1779127115.669660 read 0x00c02e thru 0x00cb84 (0xb57 bytes) 000725 1779127115.705224 trunc from 0x021a39 to 0x023a0e (0x1fd6 bytes) 000727 1779127115.710882 read 0x008b06 thru 0x00b20e (0x2709 bytes) 000728 1779127115.711602 fallocate 0x02ff82 thru 0x031a05 (0x1a84 bytes) 000731 1779127115.719278 trunc from 0x02dc85 to 0x0078a5 (0x263df bytes) 000732 1779127115.738617 mapread 0x004191 thru 0x00718c (0x2ffc bytes) 000733 1779127115.740641 write 0x00ce03 thru 0x014e17 (0x8015 bytes) 000734 1779127115.750793 MIRROR_RESYNC 5 mirrors 000735 1779127116.940269 trunc from 0x014e18 to 0x02d28f (0x18478 bytes) 000736 1779127117.067289 trunc from 0x004724 to 0x00b0a7 (0x6984 bytes) 000737 1779127117.077843 read 0x013b86 thru 0x01a4b2 (0x692d bytes) 000738 1779127117.090786 mapwrite 0x03fa65 thru 0x03ffff (0x59b bytes) 000739 1779127117.151328 mapwrite 0x028e01 thru 0x02963d (0x83d bytes) 000740 1779127117.207567 mapwrite 0x028db3 thru 0x038015 (0xf263 bytes) 000742 1779127117.291973 read 0x01c0cf thru 0x0238ea (0x781c bytes) 000743 1779127117.310814 fallocate 0x025a52 thru 0x02885c (0x2e0b bytes) 000744 1779127117.320613 trunc from 0x035970 to 0x039643 (0x3cd4 bytes) 000745 1779127117.346445 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x57f7734c (stale) Mirror 4: 0x57f7734c (stale) Mirror 6: 0x57f7734c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x57f7734c (stale) Mirror 4: 0x57f7734c (stale) Mirror 6: 0x57f7734c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000746 1779127117.789264 punch from 0x013d48 to 0x01853f (0x47f8 bytes) 000748 1779127117.794310 trunc from 0x040000 to 0x013380 (0x2cc7f bytes) 000749 1779127117.825260 fallocate 0x005b7d thru 0x00a54b (0x49cf bytes) 000750 1779127117.837562 read 0x009357 thru 0x013380 (0xa02a bytes) 000751 1779127117.846922 mapread 0x0009a5 thru 0x00c6fe (0xbd5a bytes) 000752 1779127117.869533 mapwrite 0x037b5b thru 0x03ffff (0x84a5 bytes) 000753 1779127117.924470 mapread 0x03289e thru 0x03bd11 (0x9474 bytes) 000754 1779127117.925944 mapwrite 0x034e9c thru 0x038a16 (0x3b7b bytes) 000755 1779127117.951468 mapwrite 0x01347d thru 0x014155 (0xcd9 bytes) 000757 1779127117.983497 mapwrite 0x0124da thru 0x015e40 (0x3967 bytes) 000758 1779127118.027953 trunc from 0x028561 to 0x0316b4 (0x9154 bytes) 000759 1779127118.044466 fallocate 0x025ccb thru 0x0325ba (0xc8f0 bytes) 000760 1779127118.057313 mapread 0x037efc thru 0x03ffff (0x8104 bytes) 000761 1779127118.058684 punch from 0x01e03f to 0x02dd91 (0xfd53 bytes) 000762 1779127118.072179 trunc from 0x040000 to 0x01df30 (0x220cf bytes) 000763 1779127118.092955 trunc from 0x01df31 to 0x02d85c (0xf92c bytes) 000764 1779127118.133486 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d85d): Mirror 3: 0x1da8b306 (stale) Mirror 6: 0x1da8b306 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d85d): Mirror 3: 0x1da8b306 (stale) Mirror 6: 0x1da8b306 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d85d: skipped 000765 1779127118.588950 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d85d): Mirror 6: 0x1da8b306 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d85d): Mirror 6: 0x1da8b306 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d85d: skipped 000766 1779127119.030218 trunc from 0x0061b8 to 0x00bba9 (0x59f2 bytes) 000767 1779127119.037591 trunc from 0x01ffab to 0x0297e8 (0x983e bytes) 000768 1779127119.045428 trunc from 0x02d85d to 0x03d316 (0xfaba bytes) 000769 1779127119.072272 trunc from 0x01b753 to 0x021cf5 (0x65a3 bytes) 000771 1779127119.080961 mapwrite 0x00267b thru 0x0071a8 (0x4b2e bytes) 000772 1779127119.160002 trunc from 0x03d317 to 0x02be37 (0x114df bytes) 000773 1779127119.189323 read 0x003044 thru 0x003301 (0x2be bytes) 000774 1779127119.191038 mapwrite 0x01dedc thru 0x023ba9 (0x5cce bytes) 000776 1779127119.277498 read 0x000f7d thru 0x0058f8 (0x497c bytes) 000777 1779127119.291800 trunc from 0x02be38 to 0x00a63a (0x217fd bytes) 000778 1779127119.318432 write 0x00639b thru 0x00bd6f (0x59d5 bytes) 000779 1779127119.321087 mapwrite 0x0009cd thru 0x008bb1 (0x81e5 bytes) 000781 1779127119.357859 fallocate 0x0035aa thru 0x00bcc0 (0x8717 bytes) 000782 1779127119.364278 write 0x01f5c0 thru 0x02d9e5 (0xe426 bytes) 000783 1779127119.367126 write 0x012972 thru 0x01b1dd (0x886c bytes) 000784 1779127119.381433 punch from 0x00cd79 to 0x01476f (0x79f7 bytes) 000785 1779127119.399592 mapread 0x025c9c thru 0x02d9e5 (0x7d4a bytes) 000786 1779127119.407090 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000787 1779127119.573665 trunc from 0x018f3e to 0x027051 (0xe114 bytes) 000788 1779127119.585311 trunc from 0x02d9e6 to 0x01ad31 (0x12cb4 bytes) 000789 1779127119.619165 punch from 0x00c265 to 0x0100a1 (0x3e3d bytes) 000790 1779127119.625937 mapwrite 0x0276cc thru 0x032fe8 (0xb91d bytes) 000791 1779127119.726298 trunc from 0x01ec84 to 0x029a98 (0xae15 bytes) 000792 1779127119.741281 read 0x013ebd thru 0x020538 (0xc67c bytes) 000793 1779127119.771998 trunc from 0x00eb99 to 0x0188e8 (0x9d50 bytes) 000794 1779127119.792555 write 0x0196fd thru 0x028271 (0xeb75 bytes) 000795 1779127119.808074 punch from 0x02270b to 0x027c5e (0x5554 bytes) 000796 1779127119.837968 trunc from 0x032fe9 to 0x004f21 (0x2e0c7 bytes) 000797 1779127119.866713 fallocate 0x034822 thru 0x0392c6 (0x4aa5 bytes) 000798 1779127119.876596 fallocate 0x0138d6 thru 0x01c0b6 (0x87e1 bytes) 000799 1779127119.886685 mapread 0x01a516 thru 0x01c0b6 (0x1ba1 bytes) 000800 1779127119.913834 write 0x0064b2 thru 0x00ae44 (0x4993 bytes) 000801 1779127119.950233 mapread 0x014a24 thru 0x01a692 (0x5c6f bytes) 000802 1779127119.954281 trunc from 0x013cdf to 0x01c0b6 (0x83d8 bytes) 000803 1779127119.980027 write 0x004713 thru 0x0079e1 (0x32cf bytes) 000804 1779127119.981374 punch from 0x0076f6 to 0x009300 (0x1c0b bytes) 000805 1779127120.021168 fallocate 0x023170 thru 0x023ce0 (0xb71 bytes) 000806 1779127120.031763 read 0x018405 thru 0x023ce0 (0xb8dc bytes) 000807 1779127120.057184 trunc from 0x023ce1 to 0x001700 (0x225e0 bytes) 000809 1779127120.099518 trunc from 0x000815 to 0x001700 (0xeec bytes) 000810 1779127120.106326 punch from 0x000cc5 to 0x001700 (0xa3c bytes) 000811 1779127120.111875 trunc from 0x00120d to 0x001700 (0x4f4 bytes) 000812 1779127120.116953 mapread 0x00113a thru 0x001700 (0x5c7 bytes) 000813 1779127120.124499 punch from 0x0005cb to 0x001700 (0x1136 bytes) 000814 1779127120.134284 trunc from 0x000e1c to 0x001700 (0x8e5 bytes) 000816 1779127120.140145 mapwrite 0x02588e thru 0x029af7 (0x426a bytes) 000817 1779127120.219428 punch from 0x00628d to 0x0155b3 (0xf327 bytes) 000818 1779127120.226602 trunc from 0x007bb2 to 0x016cb2 (0xf101 bytes) 000820 1779127120.235712 trunc from 0x00be4f to 0x014ff7 (0x91a9 bytes) 000822 1779127120.242873 fallocate 0x028151 thru 0x03181b (0x96cb bytes) 000823 1779127120.250160 mapwrite 0x0257ba thru 0x028c8f (0x34d6 bytes) 000824 1779127120.267820 punch from 0x00df23 to 0x01847a (0xa558 bytes) 000825 1779127120.273141 trunc from 0x00c4ef to 0x012ded (0x68ff bytes) 000826 1779127120.279182 mapwrite 0x03070c thru 0x0349b7 (0x42ac bytes) 000827 1779127120.321505 mapread 0x01be5f thru 0x01d0a4 (0x1246 bytes) 000829 1779127120.350917 fallocate 0x01c458 thru 0x02a3aa (0xdf53 bytes) 000830 1779127120.359103 mapwrite 0x01eae5 thru 0x02217e (0x369a bytes) 000831 1779127120.455787 MIRROR_EXTEND to 2 mirrors 000832 1779127121.359695 trunc from 0x00252b to 0x0062d2 (0x3da8 bytes) 000833 1779127121.445823 read 0x00618e thru 0x0115d4 (0xb447 bytes) 000834 1779127121.484482 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x349b8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349b8): Mirror 2: 0x122b623f (stale) CRC-32 checksum value for chunk [0, 0x349b8): Mirror 1: 0x122b623f Mirror 3: 0x122b623f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349b8: skipped 000835 1779127122.667484 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x349b8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349b8): Mirror 2: 0x122b623f (stale) CRC-32 checksum value for chunk [0, 0x349b8): Mirror 1: 0x122b623f Mirror 3: 0x122b623f Mirror 4: 0x122b623f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349b8: skipped 000836 1779127123.990537 read 0x01a8ae thru 0x022edc (0x862f bytes) 000837 1779127124.009255 fallocate 0x026149 thru 0x033b47 (0xd9ff bytes) 000838 1779127124.062934 write 0x00fc48 thru 0x018260 (0x8619 bytes) 000839 1779127124.086353 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x349b8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349b8): Mirror 2: 0x122b623f (stale) Mirror 3: 0x122b623f (stale) Mirror 4: 0x122b623f (stale) CRC-32 checksum value for chunk [0, 0x349b8): Mirror 1: 0x3e16aeda Mirror 5: 0x3e16aeda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349b8: skipped 000841 1779127125.303820 punch from 0x00d187 to 0x01c9b2 (0xf82c bytes) 000842 1779127125.363559 write 0x02ba55 thru 0x035466 (0x9a12 bytes) 000843 1779127125.384549 trunc from 0x01afcb to 0x024630 (0x9666 bytes) 000844 1779127125.394051 trunc from 0x013828 to 0x019ac3 (0x629c bytes) 000845 1779127125.403410 trunc from 0x02d98c to 0x035466 (0x7adb bytes) 000846 1779127125.433449 read 0x0282bb thru 0x035466 (0xd1ac bytes) 000847 1779127125.457292 read 0x02ea8d thru 0x0332d5 (0x4849 bytes) 000848 1779127125.458242 trunc from 0x035467 to 0x031d53 (0x3713 bytes) 000849 1779127125.520471 punch from 0x00e3fb to 0x010cb6 (0x28bc bytes) 000850 1779127125.532441 trunc from 0x012a18 to 0x021dea (0xf3d3 bytes) 000851 1779127125.543195 read 0x0164ef thru 0x021750 (0xb262 bytes) 000852 1779127125.586253 write 0x03676a thru 0x03ffff (0x9896 bytes) 000853 1779127125.588209 trunc from 0x01c05e to 0x01e704 (0x26a7 bytes) 000855 1779127125.602762 punch from 0x02e0c5 to 0x03d12a (0xf066 bytes) 000857 1779127125.644701 fallocate 0x0196d8 thru 0x019fc3 (0x8ec bytes) 000858 1779127125.658724 write 0x01f65a thru 0x02003e (0x9e5 bytes) 000859 1779127125.659762 mapwrite 0x03b671 thru 0x03ffff (0x498f bytes) 000860 1779127125.725771 trunc from 0x040000 to 0x02df50 (0x120af bytes) 000861 1779127125.765590 punch from 0x0097f2 to 0x0160cd (0xc8dc bytes) 000862 1779127125.772398 write 0x02f5c0 thru 0x036dde (0x781f bytes) 000863 1779127125.774800 punch from 0x030420 to 0x032236 (0x1e17 bytes) 000864 1779127125.801276 mapread 0x01a697 thru 0x025afb (0xb465 bytes) 000865 1779127125.840408 fallocate 0x026b79 thru 0x030e45 (0xa2cd bytes) 000866 1779127125.848575 trunc from 0x02540f to 0x0337f4 (0xe3e6 bytes) 000867 1779127125.864288 MIRROR_RESYNC 5 mirrors 000868 1779127127.528841 punch from 0x01b446 to 0x01f691 (0x424c bytes) 000869 1779127127.600476 read 0x0036bc thru 0x01283d (0xf182 bytes) 000870 1779127127.621413 fallocate 0x01d304 thru 0x02559b (0x8298 bytes) 000871 1779127127.628639 read 0x00b387 thru 0x00ef1d (0x3b97 bytes) 000872 1779127127.631190 trunc from 0x018a12 to 0x01bcb8 (0x32a7 bytes) 000873 1779127127.643196 MIRROR_RESYNC 5 mirrors 000874 1779127129.133002 fallocate 0x01b305 thru 0x01bf9c (0xc98 bytes) 000875 1779127129.183534 read 0x0260e1 thru 0x029e79 (0x3d99 bytes) 000876 1779127129.191352 read 0x017b02 thru 0x01fab5 (0x7fb4 bytes) 000877 1779127129.203505 write 0x000034 thru 0x00a601 (0xa5ce bytes) 000879 1779127129.206167 write 0x01b84d thru 0x02015c (0x4910 bytes) 000880 1779127129.208133 punch from 0x011e89 to 0x01ac01 (0x8d79 bytes) 000882 1779127129.214056 read 0x00e0d3 thru 0x01c44c (0xe37a bytes) 000883 1779127129.222788 read 0x005fc3 thru 0x00a86d (0x48ab bytes) 000884 1779127129.227503 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36ddf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ddf): Mirror 2: 0xc5346cd4 (stale) Mirror 3: 0xc5346cd4 (stale) Mirror 4: 0xc5346cd4 (stale) Mirror 5: 0xc5346cd4 (stale) CRC-32 checksum value for chunk [0, 0x36ddf): Mirror 1: 0x4ac1c8f Mirror 6: 0x4ac1c8f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ddf: skipped 000885 1779127130.167421 trunc from 0x036ddf to 0x01da3d (0x193a1 bytes) 000887 1779127130.313496 trunc from 0x01da3e to 0x00823b (0x15802 bytes) 000888 1779127130.366148 mapread 0x00191b thru 0x007ce0 (0x63c6 bytes) 000889 1779127130.411551 trunc from 0x00823c to 0x003195 (0x50a6 bytes) 000890 1779127130.438335 mapwrite 0x0292d2 thru 0x037ffd (0xed2c bytes) 000891 1779127130.587797 write 0x015921 thru 0x02583a (0xff1a bytes) 000892 1779127130.630590 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ffe): Mirror 2: 0x47369753 (stale) Mirror 3: 0x47369753 (stale) Mirror 4: 0x47369753 (stale) Mirror 6: 0x253fdab5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ffe): Mirror 2: 0x47369753 (stale) Mirror 3: 0x47369753 (stale) Mirror 4: 0x47369753 (stale) Mirror 6: 0x253fdab5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ffe: skipped 000893 1779127131.147334 trunc from 0x037ffe to 0x004a3d (0x335c0 bytes) 000894 1779127131.170352 trunc from 0x004a3e to 0x015f54 (0x11517 bytes) 000895 1779127131.196686 trunc from 0x015f55 to 0x03fcf9 (0x29da5 bytes) 000896 1779127131.222490 trunc from 0x03fcfa to 0x019261 (0x26a98 bytes) 000897 1779127131.241734 mapwrite 0x02a88e thru 0x02e315 (0x3a88 bytes) 000898 1779127131.303085 mapwrite 0x01150d thru 0x017762 (0x6256 bytes) 000900 1779127131.331060 fallocate 0x026eaa thru 0x027555 (0x6ac bytes) 000901 1779127131.336610 trunc from 0x015a76 to 0x016296 (0x821 bytes) 000902 1779127131.342319 fallocate 0x03cd49 thru 0x03ffff (0x32b7 bytes) 000903 1779127131.352250 MIRROR_RESYNC 5 mirrors 000905 1779127132.815270 mapread 0x0192b3 thru 0x02311a (0x9e68 bytes) 000906 1779127132.851144 punch from 0x00a86e to 0x019529 (0xecbc bytes) 000907 1779127132.899447 mapwrite 0x013124 thru 0x0207e7 (0xd6c4 bytes) 000908 1779127132.939349 mapwrite 0x02a9f8 thru 0x02eee3 (0x44ec bytes) 000909 1779127132.985927 trunc from 0x00b69f to 0x019cb1 (0xe613 bytes) 000910 1779127132.992398 write 0x023a52 thru 0x02b61e (0x7bcd bytes) 000911 1779127132.994678 write 0x020fb2 thru 0x027109 (0x6158 bytes) 000913 1779127132.997771 trunc from 0x01ac48 to 0x027ad3 (0xce8c bytes) 000914 1779127133.027932 write 0x00898d thru 0x0095fb (0xc6f bytes) 000915 1779127133.060857 write 0x01d89b thru 0x021e5c (0x45c2 bytes) 000917 1779127133.076540 punch from 0x025a6d to 0x02eee3 (0x9477 bytes) 000918 1779127133.084114 read 0x0298ce thru 0x02eee3 (0x5616 bytes) 000919 1779127133.103835 punch from 0x02022d to 0x02563d (0x5411 bytes) 000920 1779127133.130952 fallocate 0x020b42 thru 0x023ca2 (0x3161 bytes) 000921 1779127133.138697 write 0x011c93 thru 0x01b326 (0x9694 bytes) 000922 1779127133.163550 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2eee4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eee4): Mirror 2: 0x4f63ae57 (stale) Mirror 3: 0x4f63ae57 (stale) Mirror 4: 0x4f63ae57 (stale) Mirror 6: 0x4f63ae57 (stale) CRC-32 checksum value for chunk [0, 0x2eee4): Mirror 1: 0x1713c702 Mirror 7: 0x1713c702 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eee4: skipped 000923 1779127134.348813 write 0x037929 thru 0x03eba2 (0x727a bytes) 000924 1779127134.403299 fallocate 0x00123e thru 0x00274b (0x150e bytes) 000925 1779127134.409876 write 0x02ee65 thru 0x0366eb (0x7887 bytes) 000926 1779127134.422457 mapwrite 0x0042fa thru 0x007dba (0x3ac1 bytes) 000928 1779127134.484420 trunc from 0x03eba3 to 0x0318ed (0xd2b5 bytes) 000929 1779127134.579835 trunc from 0x01e263 to 0x0272a5 (0x9043 bytes) 000930 1779127134.612247 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x318ee): Mirror 2: 0x368ca88c (stale) Mirror 4: 0x368ca88c (stale) Mirror 6: 0x368ca88c (stale) Mirror 7: 0x3a49d1a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x318ee): Mirror 2: 0x368ca88c (stale) Mirror 4: 0x368ca88c (stale) Mirror 6: 0x368ca88c (stale) Mirror 7: 0x3a49d1a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x318ee: skipped 000931 1779127135.210080 punch from 0x02ca01 to 0x0318ed (0x4eed bytes) 000932 1779127135.219698 trunc from 0x008c30 to 0x0180da (0xf4ab bytes) 000933 1779127135.228484 fallocate 0x01dde6 thru 0x0248ef (0x6b0a bytes) 000934 1779127135.236309 write 0x033566 thru 0x0340a9 (0xb44 bytes) 000935 1779127135.252376 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x340aa): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340aa): Mirror 2: 0x58177b2f (stale) Mirror 4: 0x58177b2f (stale) Mirror 6: 0x58177b2f (stale) Mirror 7: 0x5dd1fbcc (stale) CRC-32 checksum value for chunk [0, 0x340aa): Mirror 1: 0xd5ae2cdb Mirror 8: 0xd5ae2cdb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340aa: skipped 000936 1779127136.632107 MIRROR_RESYNC 6 mirrors 000937 1779127138.462846 fallocate 0x03b318 thru 0x03ffff (0x4ce8 bytes) 000938 1779127138.507369 punch from 0x002dd5 to 0x00df7b (0xb1a7 bytes) 000939 1779127138.516340 trunc from 0x017242 to 0x019812 (0x25d1 bytes) 000940 1779127138.522503 mapwrite 0x009567 thru 0x016933 (0xd3cd bytes) 000941 1779127138.637102 trunc from 0x006378 to 0x0141fc (0xde85 bytes) 000943 1779127138.643243 mapwrite 0x01a0f4 thru 0x024db8 (0xacc5 bytes) 000944 1779127138.685830 mapwrite 0x00d52f thru 0x014528 (0x6ffa bytes) 000945 1779127138.726138 trunc from 0x0340aa to 0x0042c0 (0x2fde9 bytes) 000946 1779127138.757769 mapwrite 0x02e534 thru 0x0355db (0x70a8 bytes) 000947 1779127138.850406 mapwrite 0x025807 thru 0x02b4a7 (0x5ca1 bytes) 000948 1779127138.883304 fallocate 0x020701 thru 0x0259b5 (0x52b5 bytes) 000949 1779127138.888474 trunc from 0x0355dc to 0x02071e (0x14ebd bytes) 000950 1779127138.913815 write 0x03b136 thru 0x03ffff (0x4eca bytes) 000951 1779127138.915311 read 0x03f2b0 thru 0x03ffff (0xd50 bytes) 000952 1779127138.917182 trunc from 0x00a1dc to 0x013e9a (0x9cbf bytes) 000953 1779127138.923505 trunc from 0x040000 to 0x0013ce (0x3ec31 bytes) 000954 1779127138.949581 punch from 0x000be9 to 0x0013ce (0x7e6 bytes) 000955 1779127138.957149 mapread 0x000a23 thru 0x0013ce (0x9ac bytes) 000956 1779127138.964853 mapwrite 0x02c0ed thru 0x03059e (0x44b2 bytes) 000957 1779127139.023393 punch from 0x00e15c to 0x012402 (0x42a7 bytes) 000958 1779127139.035379 read 0x01f2f9 thru 0x02b8e5 (0xc5ed bytes) 000959 1779127139.047324 read 0x02b191 thru 0x03059e (0x540e bytes) 000961 1779127139.048451 mapread 0x019fac thru 0x020de8 (0x6e3d bytes) 000962 1779127139.049812 trunc from 0x03059f to 0x01edd7 (0x117c7 bytes) 000963 1779127139.077333 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1edd8): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1edd8): Mirror 2: 0x43e55308 (stale) Mirror 4: 0x43e55308 (stale) Mirror 6: 0x43e55308 (stale) Mirror 7: 0x43e55308 (stale) Mirror 8: 0x43e55308 (stale) CRC-32 checksum value for chunk [0, 0x1edd8): Mirror 1: 0xd46907e7 Mirror 9: 0xd46907e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1edd8: skipped 000964 1779127139.886722 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1edd8): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1edd8): Mirror 2: 0x43e55308 (stale) Mirror 4: 0x43e55308 (stale) Mirror 6: 0x43e55308 (stale) Mirror 7: 0x43e55308 (stale) Mirror 8: 0x43e55308 (stale) CRC-32 checksum value for chunk [0, 0x1edd8): Mirror 1: 0xd46907e7 Mirror 9: 0xd46907e7 Mirror 10: 0xd46907e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1edd8: skipped 000965 1779127140.866677 mapread 0x00ec29 thru 0x017e5e (0x9236 bytes) 000966 1779127140.896869 read 0x018cd1 thru 0x01edd7 (0x6107 bytes) 000967 1779127140.906400 mapwrite 0x028e6a thru 0x0292e3 (0x47a bytes) 000968 1779127141.107605 write 0x01d603 thru 0x026510 (0x8f0e bytes) 000970 1779127141.146151 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x292e4): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x292e4): Mirror 2: 0x45ec18d2 (stale) Mirror 4: 0x45ec18d2 (stale) Mirror 6: 0x45ec18d2 (stale) Mirror 7: 0x45ec18d2 (stale) Mirror 8: 0x45ec18d2 (stale) Mirror 9: 0xf027cc78 (stale) Mirror 10: 0xf027cc78 (stale) CRC-32 checksum value for chunk [0, 0x292e4): Mirror 1: 0xdf3be24 Mirror 11: 0xdf3be24 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x292e4: skipped 000971 1779127142.572431 mapwrite 0x005629 thru 0x010bc1 (0xb599 bytes) 000972 1779127142.724795 MIRROR_RESYNC 9 mirrors 000973 1779127145.954822 mapread 0x011e2a thru 0x01ec5c (0xce33 bytes) 000974 1779127145.991510 punch from 0x00f2d4 to 0x011328 (0x2055 bytes) 000976 1779127146.035676 mapread 0x018852 thru 0x02528a (0xca39 bytes) 000977 1779127146.058192 trunc from 0x0191b9 to 0x01f227 (0x606f bytes) 000978 1779127146.064347 fallocate 0x025b6c thru 0x031367 (0xb7fc bytes) 000979 1779127146.070325 fallocate 0x007afb thru 0x017269 (0xf76f bytes) 000980 1779127146.077201 trunc from 0x01530c to 0x016e2c (0x1b21 bytes) 000981 1779127146.082461 mapwrite 0x03a1c8 thru 0x03ffff (0x5e38 bytes) 000982 1779127146.132393 read 0x038067 thru 0x03ffff (0x7f99 bytes) 000983 1779127146.133262 trunc from 0x040000 to 0x02a0b5 (0x15f4a bytes) 000985 1779127146.152969 mapwrite 0x00d0e7 thru 0x013e7c (0x6d96 bytes) 000986 1779127146.190249 mapwrite 0x00a95f thru 0x018fd0 (0xe672 bytes) 000987 1779127146.222256 write 0x0073cc thru 0x012276 (0xaeab bytes) 000988 1779127146.229099 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2a0b6): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a0b6): Mirror 2: 0x2ad1d3c3 (stale) Mirror 4: 0x2ad1d3c3 (stale) Mirror 6: 0x2ad1d3c3 (stale) Mirror 7: 0x2ad1d3c3 (stale) Mirror 8: 0x2ad1d3c3 (stale) Mirror 9: 0x2ad1d3c3 (stale) Mirror 10: 0x2ad1d3c3 (stale) Mirror 11: 0x2ad1d3c3 (stale) CRC-32 checksum value for chunk [0, 0x2a0b6): Mirror 1: 0xee396255 Mirror 12: 0xee396255 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a0b6: skipped 000989 1779127148.051016 mapwrite 0x02d4ad thru 0x034914 (0x7468 bytes) 000990 1779127148.213326 punch from 0x028e75 to 0x032ee8 (0xa074 bytes) 000991 1779127148.221387 trunc from 0x02641a to 0x028a3c (0x2623 bytes) 000992 1779127148.230844 mapwrite 0x00198b thru 0x0024d5 (0xb4b bytes) 000993 1779127148.313983 mapread 0x028009 thru 0x02ca64 (0x4a5c bytes) 000994 1779127148.345832 punch from 0x02358e to 0x02dde7 (0xa85a bytes) 000995 1779127148.367188 mapwrite 0x014383 thru 0x0192bf (0x4f3d bytes) 000997 1779127148.441853 trunc from 0x034394 to 0x034914 (0x581 bytes) 000998 1779127148.466472 fallocate 0x01e182 thru 0x02bb61 (0xd9e0 bytes) 000999 1779127148.476594 trunc from 0x034915 to 0x015fbf (0x1e955 bytes) 001000 1779127148.519825 fallocate 0x035d6b thru 0x03ffff (0xa295 bytes) All operations completed A-OK! PASS 70a (96s) == sanity-flr test 71: check mirror extend parent fid ==== 13:59:13 (1779127153) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x59:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 13:59:23 (1779127163) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.274566 s, 7.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804849 s, 13.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c4:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 13:59:31 (1779127171) Starting client: oleg636-client.virtnet: -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre2 Starting client: oleg636-client.virtnet: -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7918560 .. Extending file size to 1877056 .. Extending file size to 8139776 .. Extending file size to 8238848 .. Extending file size to 8424736 .. Extending file size to 8496160 .. Extending file size to 2263936 .. Extending file size to 3753664 .. Extending file size to 4428800 .. Extending file size to 6420704 .. Extending file size to 6874112 .. Extending file size to 7084832 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8822048 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9095616 .. Extending file size to 9185664 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 60853 60855 60857 60858 60859 192.168.206.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg636-client.virtnet /mnt/lustre2 (opts:) 192.168.206.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg636-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:00:53 (1779127253) Starting client: oleg636-client.virtnet: -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre2 Starting client: oleg636-client.virtnet: -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3866176 .. Extending file size to 1858752 .. Extending file size to 8289376 .. Extending file size to 8631744 .. Extending file size to 8558720 .. Extending file size to 8799680 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9082240 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9272608 .. Extending file size to 8807552 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62404 62405 62406 62408 62409 done 192.168.206.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg636-client.virtnet /mnt/lustre2 (opts:) 192.168.206.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg636-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:02:12 (1779127332) Starting client: oleg636-client.virtnet: -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1268876 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1960 1269344 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18948 3588072 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24792 3582228 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43740 7170300 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2428 1268876 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1960 1269344 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18948 3588072 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24792 3582228 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43740 7170300 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103083 s, 20.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.275107 s, 7.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe71a72a3 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg636-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:02:34 (1779127354) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166721 s, 12.6 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 14:02:42 (1779127362) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a6:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d9:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d9:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:02:49 (1779127369) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100923 s, 20.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:02:57 (1779127377) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102535 s, 20.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2db:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2db:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2db:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:03:06 (1779127386) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151135 s, 13.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dd:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:03:40 (1779127420) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg636-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124203 s, 8.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x50:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x50:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:04:37 (1779127477) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg636-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114114 s, 18.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:05:36 (1779127536) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg636-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg636-server: Pool lustre.test_204f created Adding targets to pool oleg636-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg636-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107028 s, 19.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg636-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg636-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg636-server: Pool lustre.test_204f destroyed PASS 204f (66s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:06:41 (1779127601) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:06:49 (1779127609) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:06:56 (1779127616) oleg636-server: Pool lustre.test_206 created Adding targets to pool oleg636-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg636-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg636-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg636-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg636-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:07:25 (1779127645) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629885 s, 16.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486291 s, 21.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4d24880b Mirror 3: 0x4d24880b PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:07:39 (1779127659) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:07:42 (1779127662) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:07:46 (1779127666) Starting client: oleg636-client.virtnet: -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0722962 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2530000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x6c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x6c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x6c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2600000 Read this (4 bytes): BBBB 192.168.206.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg636-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:08:26 (1779127706) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0153849 s, 532 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:08:33 (1779127713) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536823 s, 19.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1c5 lmm_fid: [0x200000405:0x1c5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1c5 lmm_fid: [0x200000405:0x1c5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e9:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611589 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-18 14:08:34.000000000 -0400 Modify: 2026-05-18 14:08:34.000000000 -0400 Change: 2026-05-18 14:08:35.000000000 -0400 Birth: 2026-05-18 14:08:34.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:08:40 (1779127720) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342865 s, 30.6 MB/s oleg636-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg636-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg636-server: O/280000401/d7/2e7: File not found by ext2_lookup PASS 210b (19s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:08:59 (1779127739) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.235074 s, 174 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0479752 s, 85.4 kB/s size after second write -rw-r--r-- 1 root root 4096 May 18 14:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ca:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 18 14:09 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:09:07 (1779127747) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:09:14 (1779127754) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1804 sec =========== 14:09:21 (1779127761) === sanity-flr: start cleanup 14:09:23 (1779127763) === === sanity-flr: finish cleanup 14:09:32 (1779127772) ===