-----============= acceptance-small: sanity-flr ============----- Sat Sep 5 17:15:59 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg322-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 17:16:18 (1788642978) === oleg322-client.virtnet: executing check_config_client /mnt/lustre oleg322-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg322-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8f52c8042800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f52c8042800.idle_timeout=debug disable quota as required oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 17:17:04 (1788643024) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8210] == sanity-flr test 0a: lfs mirror create with -N option == 17:17:07 (1788643027) PASS 0a (22s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:17:28 (1788643048) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:17:32 (1788643052) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:17:36 (1788643056) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:18:02 (1788643082) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:18:05 (1788643085) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 17:18:09 (1788643089) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8f52c8042800.stats=clear osc.lustre-OST0001-osc-ffff8f52c8042800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:18:18 (1788643098) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:18:32 (1788643112) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:18:43 (1788643123) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 17:18:47 (1788643127) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:18:55 (1788643135) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 17:19:06 (1788643146) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:19:13 (1788643153) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:19:21 (1788643161) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:19:30 (1788643170) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:19:41 (1788643181) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:19:48 (1788643188) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.10616 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141675 s, 7.4 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:19:58 (1788643198) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.24532 s, 29.9 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 17:20:05 (1788643205) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:20:14 (1788643214) Stopping /mnt/lustre-ost1 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f52c8042800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f52c8042800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg322-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg322-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg322-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg322-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg322-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg322-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg322-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg322-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg322-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg322-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg322-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg322-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg322-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg322-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg322-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg322-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg322-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg322-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg322-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg322-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg322-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg322-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg322-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg322-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg322-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f52c8042800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f52c8042800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg322-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (129s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:22:23 (1788643343) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.482684 s, 19.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f52c8042800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f52c8042800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg322-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:23:03 (1788643383) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f52c8042800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f52c8042800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg322-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f52c8042800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f52c8042800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg322-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:24:21 (1788643461) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.08098 s, 25.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 5 17:24 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.687s user 0m0.000s sys 0m0.179s Stopping /mnt/lustre-ost1 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f52c8042800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f52c8042800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.887s user 0m0.001s sys 0m0.193s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg322-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f52c8042800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f52c8042800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.336s user 0m0.001s sys 0m0.179s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg322-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f52c8042800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f52c8042800.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:25:54 (1788643554) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:25:58 (1788643558) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:26:01 (1788643561) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 17:26:05 (1788643565) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 17:26:12 (1788643572) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.710667 s, 17.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:26:25 (1788643585) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216076 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249967 s, 41.9 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:26:33 (1788643593) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:26:42 (1788643602) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000629974 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00169803 s, 603 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00311737 s, 328 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00260926 s, 392 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 17:26:50 (1788643610) create test files with size 15730304 .. 282ffeb01a3d5cc3d802a0da09d70f5d - a780f9b13d159fd94cdca07e57a5c431 - e4bd246e75d8c1c61a60f55efdc83269 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg322-client.virtnet /mnt/lustre (opts:) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 17:27:13 (1788643633) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:27:28 (1788643648) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:27:36 (1788643656) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068235 s, 15.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xe0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xe1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x9d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718497 s, 14.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x9e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x9f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xe2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 17:27:46 (1788643666) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0933998 s, 22.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.381954 s, 11.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20781 s, 10.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 17:28:02 (1788643682) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 17:28:05 (1788643685) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:28:08 (1788643688) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424154 s, 24.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 17:28:15 (1788643695) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.322125 s, 9.8 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:28:25 (1788643705) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:28:35 (1788643715) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.290386 s, 36.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg322-client.virtnet /mnt/lustre (opts:) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:28:46 (1788643726) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8f52c7831000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.287708 s, 36.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg322-client.virtnet /mnt/lustre (opts:) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8f52c9184800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:29:01 (1788643741) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:29:47 (1788643787) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:30:27 (1788643827) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 17:30:30 (1788643830) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.228473 s, 13.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.169431 s, 18.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x3f11e9dd (nosync,stale) Mirror 2: 0x3f11e9dd (nosync,stale) ** verify mirror 2 doesn't change original checksum: df5dbec0243fcddd02546014bf3d8690 - mirror 1 checksum: 2416b52a393694aea42f2b9adf0f13c1 - mirror 2 checksum: df5dbec0243fcddd02546014bf3d8690 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T17:30:33-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T17:30:33-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: df5dbec0243fcddd02546014bf3d8690 - mirror 1 checksum: 2416b52a393694aea42f2b9adf0f13c1 - mirror 2 checksum: 2416b52a393694aea42f2b9adf0f13c1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T17:30:33-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T17:30:33-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 17:30:42 (1788643842) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436076 s, 24.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:30:51 (1788643851) Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 17:31:03 (1788643863) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 17:31:28 (1788643888) targets are mounted 17:31:28 (1788643888) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614405 s, 17.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb6:0x0] } PASS 50A (57s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:31:48 (1788643908) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.116667 s, 176 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0394006 s, 520 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:32:00 (1788643920) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0729224 s, 281 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00931387 s, 2.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:32:12 (1788643932) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00866688 s, 1.9 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:32:20 (1788643940) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0383147 s, 535 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:32:29 (1788643949) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0143823 s, 285 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0628691 s, 65.2 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:32:38 (1788643958) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:33:02 (1788643982) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:33:30 (1788644010) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:33:57 (1788644037) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:34:24 (1788644064) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (11s) == sanity-flr test 70: mirror create and split race ====== 17:34:36 (1788644076) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57582 mirror split pid 57583 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57582 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57583 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 17:35:44 (1788644144) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 4146 000001 1788644147.259404 mapwrite 0x0311b6 thru 0x0384f0 (0x733b bytes) 000002 1788644147.395737 mapwrite 0x005396 thru 0x0115c7 (0xc232 bytes) 000003 1788644147.495847 write 0x03bec2 thru 0x03ffff (0x413e bytes) zero_range to largest ever: 0xfd84 000004 1788644147.497030 trunc from 0x0017c1 to 0x00fd83 (0xe5c3 bytes) 000005 1788644147.506403 mapread 0x03ffd9 thru 0x03ffff (0x27 bytes) 000007 1788644147.510923 write 0x00a5cf thru 0x0145bf (0x9ff1 bytes) 000008 1788644147.517400 write 0x032929 thru 0x03744e (0x4b26 bytes) 000009 1788644147.519354 mapread 0x02fb55 thru 0x03aeff (0xb3ab bytes) 000010 1788644147.525537 trunc from 0x040000 to 0x00e3b1 (0x31c4e bytes) 000011 1788644147.540287 mapread 0x0045f5 thru 0x0085d7 (0x3fe3 bytes) 000012 1788644147.549479 mapwrite 0x02e51a thru 0x032057 (0x3b3e bytes) 000013 1788644147.604457 mapwrite 0x00c290 thru 0x00dca0 (0x1a11 bytes) 000014 1788644147.621477 write 0x03fa28 thru 0x03ffff (0x5d8 bytes) 000015 1788644147.623858 punch from 0x022491 to 0x02fd6b (0xd8db bytes) 000016 1788644147.635324 write 0x00ac41 thru 0x012b90 (0x7f50 bytes) 000018 1788644147.648174 write 0x014f28 thru 0x014f5c (0x35 bytes) 000019 1788644147.667305 write 0x028501 thru 0x02d40c (0x4f0c bytes) 000020 1788644147.689286 read 0x005464 thru 0x00ad71 (0x590e bytes) 000021 1788644147.690062 trunc from 0x00028c to 0x00c448 (0xc1bd bytes) 000022 1788644147.726185 read 0x03cb5e thru 0x03ffff (0x34a2 bytes) fallocating to largest ever: 0x2084f 000023 1788644147.763033 fallocate 0x0184c3 thru 0x02084e (0x838c bytes) 000025 1788644147.777207 read 0x013833 thru 0x0155c1 (0x1d8f bytes) fallocating to largest ever: 0x2b461 000026 1788644147.777953 fallocate 0x02586c thru 0x02b460 (0x5bf5 bytes) fallocating to largest ever: 0x40000 000027 1788644147.789576 fallocate 0x03f1cc thru 0x03ffff (0xe34 bytes) 000028 1788644147.796462 fallocate 0x033d2d thru 0x037489 (0x375d bytes) 000029 1788644147.808053 trunc from 0x040000 to 0x03a6d4 (0x592b bytes) 000031 1788644147.865111 trunc from 0x03a6d5 to 0x031201 (0x94d3 bytes) 000032 1788644147.883809 fallocate 0x02686e thru 0x026932 (0xc5 bytes) 000033 1788644147.889533 fallocate 0x038a62 thru 0x039e1a (0x13b9 bytes) 000035 1788644147.899462 trunc from 0x00d55e to 0x01c93b (0xf3de bytes) 000036 1788644147.906816 mapread 0x02adbb thru 0x0321fd (0x7443 bytes) 000037 1788644147.915580 trunc from 0x039e1b to 0x018f94 (0x20e86 bytes) 000038 1788644147.929029 fallocate 0x02e839 thru 0x03dbbb (0xf383 bytes) 000039 1788644147.935601 mapwrite 0x022c66 thru 0x030b96 (0xdf31 bytes) 000041 1788644148.049589 fallocate 0x03604c thru 0x038684 (0x2639 bytes) 000042 1788644148.055094 mapwrite 0x01fbe0 thru 0x024435 (0x4856 bytes) 000043 1788644148.086233 mapread 0x02c4f9 thru 0x038684 (0xc18c bytes) 000044 1788644148.144220 write 0x024a23 thru 0x031e98 (0xd476 bytes) 000045 1788644148.153364 trunc from 0x038685 to 0x027e9f (0x107e5 bytes) 000046 1788644148.183535 mapread 0x00cdbe thru 0x01b265 (0xe4a8 bytes) 000047 1788644148.227802 mapwrite 0x00762c thru 0x0101c8 (0x8b9d bytes) 000049 1788644148.275617 MIRROR_EXTEND to 2 mirrors 000050 1788644149.063034 read 0x01a669 thru 0x01c216 (0x1bae bytes) 000051 1788644149.084170 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000052 1788644149.224386 trunc from 0x027ea0 to 0x02f864 (0x79c5 bytes) 000053 1788644149.297889 mapwrite 0x030ba6 thru 0x03771f (0x6b7a bytes) 000054 1788644149.361389 mapread 0x0288f3 thru 0x03771f (0xee2d bytes) 000055 1788644149.377274 MIRROR_EXTEND to 2 mirrors 000056 1788644150.345275 fallocate 0x00aa47 thru 0x00c3b6 (0x1970 bytes) 000057 1788644150.450028 mapread 0x000d9e thru 0x00e35a (0xd5bd bytes) 000058 1788644150.508109 trunc from 0x00ec87 to 0x016b74 (0x7eee bytes) 000059 1788644150.531852 read 0x017246 thru 0x0238af (0xc66a bytes) 000061 1788644150.574942 MIRROR_RESYNC 2 mirrors 000062 1788644151.393522 punch from 0x02b25d to 0x036fec (0xbd90 bytes) 000063 1788644151.442249 punch from 0x00e637 to 0x01a674 (0xc03e bytes) 000065 1788644151.452681 mapread 0x01637d thru 0x01dfac (0x7c30 bytes) 000066 1788644151.473851 punch from 0x035686 to 0x03771f (0x209a bytes) 000067 1788644151.479679 trunc from 0x037720 to 0x0217e0 (0x15f3f bytes) 000068 1788644151.518173 trunc from 0x0172f2 to 0x0217e0 (0xa4ef bytes) 000069 1788644151.525461 fallocate 0x007ee6 thru 0x009990 (0x1aab bytes) 000070 1788644151.530456 punch from 0x009c74 to 0x013288 (0x9615 bytes) 000071 1788644151.544307 read 0x00330d thru 0x00c769 (0x945d bytes) 000072 1788644151.588215 fallocate 0x03d2c8 thru 0x03ffff (0x2d38 bytes) 000073 1788644151.595423 write 0x01b49e thru 0x02a3fa (0xef5d bytes) 000074 1788644151.606768 trunc from 0x01aa63 to 0x01ebe0 (0x417e bytes) 000075 1788644151.625437 trunc from 0x040000 to 0x01015d (0x2fea2 bytes) 000076 1788644151.642577 trunc from 0x01015e to 0x00c69e (0x3abf bytes) 000078 1788644151.672127 punch from 0x000a5a to 0x004b9b (0x4142 bytes) 000079 1788644151.682641 mapread 0x0055c3 thru 0x007075 (0x1ab3 bytes) 000080 1788644151.686193 punch from 0x000dc5 to 0x00c2b4 (0xb4f0 bytes) 000081 1788644151.693476 punch from 0x005f16 to 0x00c69e (0x6789 bytes) 000082 1788644151.697200 read 0x005654 thru 0x00adfe (0x57ab bytes) 000083 1788644151.715813 read 0x00ba7f thru 0x00c69e (0xc20 bytes) 000084 1788644151.716595 punch from 0x0093b1 to 0x00c69e (0x32ee bytes) 000085 1788644151.728639 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000086 1788644151.975220 trunc from 0x00419b to 0x00c69e (0x8504 bytes) 000087 1788644151.981151 mapread 0x0074b9 thru 0x009feb (0x2b33 bytes) 000088 1788644152.005843 trunc from 0x00aee5 to 0x00c69e (0x17ba bytes) 000090 1788644152.011157 mapwrite 0x0163f0 thru 0x01907f (0x2c90 bytes) 000091 1788644152.070260 read 0x0038c6 thru 0x009dc1 (0x64fc bytes) 000092 1788644152.073950 write 0x0045ac thru 0x0046a2 (0xf7 bytes) 000093 1788644152.074997 trunc from 0x019080 to 0x01c57f (0x3500 bytes) 000094 1788644152.098890 trunc from 0x00098b to 0x00e860 (0xded6 bytes) 000095 1788644152.110952 trunc from 0x01c580 to 0x029f5f (0xd9e0 bytes) 000096 1788644152.126562 read 0x00645b thru 0x00f308 (0x8eae bytes) 000097 1788644152.159859 punch from 0x01e1cf to 0x027c98 (0x9aca bytes) 000098 1788644152.169612 MIRROR_EXTEND to 2 mirrors 000099 1788644152.950150 write 0x03d54f thru 0x03ffff (0x2ab1 bytes) 000100 1788644153.016323 read 0x012b4c thru 0x01650e (0x39c3 bytes) 000101 1788644153.030654 read 0x0155aa thru 0x0233a3 (0xddfa bytes) 000102 1788644153.045990 write 0x03e2df thru 0x03ffff (0x1d21 bytes) 000103 1788644153.060459 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfc62be7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x425d2952 Mirror 3: 0x425d2952 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000104 1788644154.096162 mapread 0x0022f4 thru 0x011446 (0xf153 bytes) 000105 1788644154.162757 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000106 1788644154.602410 punch from 0x0210a3 to 0x029f2d (0x8e8b bytes) 000107 1788644154.663877 mapwrite 0x00cf56 thru 0x019257 (0xc302 bytes) 000108 1788644154.737220 trunc from 0x02c90b to 0x02dc62 (0x1358 bytes) 000109 1788644154.747373 trunc from 0x009d79 to 0x00e78a (0x4a12 bytes) 000110 1788644154.759953 MIRROR_RESYNC 2 mirrors 000112 1788644155.819834 read 0x0323d3 thru 0x03246b (0x99 bytes) 000113 1788644155.829918 fallocate 0x00e17f thru 0x00f654 (0x14d6 bytes) 000114 1788644155.882108 MIRROR_RESYNC 2 mirrors 000115 1788644156.498907 mapread 0x019a1e thru 0x01a3fb (0x9de bytes) 000116 1788644156.537321 mapwrite 0x03e948 thru 0x03ffff (0x16b8 bytes) 000117 1788644156.693293 trunc from 0x016782 to 0x017379 (0xbf8 bytes) 000118 1788644156.719301 trunc from 0x040000 to 0x038bd4 (0x742b bytes) 000119 1788644156.776594 punch from 0x0094cf to 0x0148ac (0xb3de bytes) 000121 1788644156.793189 read 0x022595 thru 0x02f32d (0xcd99 bytes) 000122 1788644156.828307 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38bd5): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38bd5): Mirror 3: 0x63a06dbb (stale) CRC-32 checksum value for chunk [0, 0x38bd5): Mirror 1: 0x94609268 Mirror 4: 0x94609268 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38bd5: skipped 000123 1788644157.759949 trunc from 0x01ea6d to 0x02d9dd (0xef71 bytes) 000124 1788644157.863540 trunc from 0x01ecc8 to 0x021ccd (0x3006 bytes) 000125 1788644157.876534 MIRROR_RESYNC 3 mirrors 000126 1788644158.706461 fallocate 0x0188a0 thru 0x01b7f8 (0x2f59 bytes) 000128 1788644158.750337 punch from 0x01f4b2 to 0x02f23e (0xfd8d bytes) 000129 1788644158.757202 trunc from 0x038bd5 to 0x016a02 (0x221d2 bytes) 000130 1788644158.806302 trunc from 0x016a03 to 0x02ffdd (0x195db bytes) 000131 1788644158.834380 trunc from 0x01ffa2 to 0x02d9d4 (0xda33 bytes) 000132 1788644158.850403 mapread 0x005770 thru 0x009812 (0x40a3 bytes) 000133 1788644158.901558 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ffde): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffde): Mirror 3: 0xcb1b3947 (stale) Mirror 4: 0xcb1b3947 (stale) CRC-32 checksum value for chunk [0, 0x2ffde): Mirror 1: 0xf81c6596 Mirror 5: 0xf81c6596 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffde: skipped 000134 1788644160.198038 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ffde): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffde): Mirror 3: 0xcb1b3947 (stale) Mirror 4: 0xcb1b3947 (stale) CRC-32 checksum value for chunk [0, 0x2ffde): Mirror 1: 0xf81c6596 Mirror 5: 0xf81c6596 Mirror 6: 0xf81c6596 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffde: skipped 000135 1788644161.456420 write 0x036853 thru 0x03ffff (0x97ad bytes) 000136 1788644161.505466 trunc from 0x002e60 to 0x010767 (0xd908 bytes) 000137 1788644161.516078 punch from 0x01c380 to 0x025e82 (0x9b03 bytes) 000138 1788644161.529227 trunc from 0x0077f5 to 0x011428 (0x9c34 bytes) 000139 1788644161.544010 mapwrite 0x01afde thru 0x02a2bf (0xf2e2 bytes) 000140 1788644161.690167 trunc from 0x040000 to 0x024556 (0x1baa9 bytes) 000141 1788644161.729100 punch from 0x003b81 to 0x012388 (0xe808 bytes) 000142 1788644161.741092 mapread 0x00d4fe thru 0x00f3f2 (0x1ef5 bytes) 000143 1788644161.764913 mapwrite 0x0028d7 thru 0x010092 (0xd7bc bytes) 000144 1788644161.811211 mapread 0x00fbc9 thru 0x01848b (0x88c3 bytes) 000145 1788644161.813332 mapwrite 0x0143e5 thru 0x02173a (0xd356 bytes) 000146 1788644161.881665 MIRROR_RESYNC 5 mirrors 000147 1788644163.092649 write 0x00eaa3 thru 0x01d36b (0xe8c9 bytes) 000148 1788644163.149305 mapwrite 0x002437 thru 0x00f44c (0xd016 bytes) 000149 1788644163.243563 read 0x00954d thru 0x018e1c (0xf8d0 bytes) 000151 1788644163.249664 write 0x01b591 thru 0x01ce0a (0x187a bytes) 000152 1788644163.255855 mapwrite 0x01728e thru 0x01e881 (0x75f4 bytes) 000153 1788644163.318542 write 0x002641 thru 0x006738 (0x40f8 bytes) 000154 1788644163.322621 mapread 0x0112d8 thru 0x020ab2 (0xf7db bytes) 000155 1788644163.325319 write 0x00f335 thru 0x0188bf (0x958b bytes) 000156 1788644163.327462 punch from 0x02008e to 0x024556 (0x44c9 bytes) 000157 1788644163.341142 fallocate 0x037a8e thru 0x03dc26 (0x6199 bytes) 000158 1788644163.350342 fallocate 0x02c5e3 thru 0x03183b (0x5259 bytes) 000159 1788644163.358241 trunc from 0x03dc27 to 0x01aa38 (0x231ee bytes) 000160 1788644163.388232 trunc from 0x01aa39 to 0x011c01 (0x8e37 bytes) 000161 1788644163.413171 mapwrite 0x02c767 thru 0x03a5b7 (0xde51 bytes) 000162 1788644163.539900 fallocate 0x02b9cc thru 0x03432c (0x8961 bytes) 000163 1788644163.571785 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a5b8): Mirror 3: 0xf0c66045 (stale) Mirror 5: 0xf0c66045 (stale) Mirror 6: 0xf0c66045 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5b8): Mirror 3: 0xf0c66045 (stale) Mirror 5: 0xf0c66045 (stale) Mirror 6: 0xf0c66045 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5b8: skipped 000164 1788644164.206228 trunc from 0x03a5b8 to 0x025131 (0x15486 bytes) 000165 1788644164.227145 trunc from 0x0106c0 to 0x01b096 (0xa9d7 bytes) 000166 1788644164.234106 fallocate 0x00ce55 thru 0x01478a (0x7936 bytes) 000167 1788644164.250017 MIRROR_RESYNC 4 mirrors 000168 1788644165.332094 fallocate 0x01aff0 thru 0x01de41 (0x2e52 bytes) 000169 1788644165.377240 fallocate 0x001442 thru 0x001af6 (0x6b5 bytes) 000170 1788644165.383134 write 0x01d645 thru 0x025b0c (0x84c8 bytes) 000171 1788644165.392240 write 0x027740 thru 0x036e44 (0xf705 bytes) 000172 1788644165.394085 mapread 0x0091da thru 0x00c6c7 (0x34ee bytes) 000173 1788644165.414343 trunc from 0x036e45 to 0x0043d3 (0x32a71 bytes) 000174 1788644165.439394 write 0x015e01 thru 0x0173ed (0x15ed bytes) 000175 1788644165.441460 read 0x00d095 thru 0x012111 (0x507d bytes) 000176 1788644165.451132 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x173ee): Mirror 5: 0xed190ffa (stale) Mirror 6: 0xed190ffa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x173ee): Mirror 5: 0xed190ffa (stale) Mirror 6: 0xed190ffa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x173ee: skipped 000177 1788644165.890296 mapread 0x01568f thru 0x0173ed (0x1d5f bytes) 000178 1788644165.923745 mapread 0x00abd5 thru 0x00e32e (0x375a bytes) 000179 1788644165.933406 fallocate 0x02e577 thru 0x036ca0 (0x872a bytes) 000180 1788644165.949144 mapread 0x007752 thru 0x014640 (0xceef bytes) 000181 1788644165.955298 trunc from 0x0173ee to 0x02e8e1 (0x174f4 bytes) 000182 1788644165.987478 trunc from 0x02e8e2 to 0x024cbd (0x9c24 bytes) 000183 1788644166.012114 mapwrite 0x0247d7 thru 0x02e511 (0x9d3b bytes) 000184 1788644166.064289 punch from 0x0039f6 to 0x012276 (0xe881 bytes) 000185 1788644166.070289 fallocate 0x017248 thru 0x01cc16 (0x59cf bytes) 000186 1788644166.083771 fallocate 0x0327d1 thru 0x0359b9 (0x31e9 bytes) 000188 1788644166.093897 trunc from 0x02e512 to 0x0398f2 (0xb3e1 bytes) 000189 1788644166.134075 punch from 0x0032c4 to 0x00423d (0xf7a bytes) 000190 1788644166.145544 fallocate 0x002f25 thru 0x00b287 (0x8363 bytes) 000191 1788644166.165950 trunc from 0x00b21d to 0x017523 (0xc307 bytes) 000192 1788644166.188258 mapwrite 0x029a92 thru 0x03485a (0xadc9 bytes) 000193 1788644166.272934 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x398f3): Mirror 6: 0xea47724b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x398f3): Mirror 6: 0xea47724b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x398f3: skipped 000194 1788644166.686750 trunc from 0x0398f3 to 0x031615 (0x82dd bytes) 000195 1788644166.711411 trunc from 0x014737 to 0x01e267 (0x9b31 bytes) 000198 1788644166.723621 trunc from 0x031616 to 0x023c11 (0xda04 bytes) 000199 1788644166.759559 punch from 0x0077cf to 0x00981c (0x204e bytes) 000200 1788644166.766387 mapread 0x013ed0 thru 0x01886a (0x499b bytes) 000201 1788644166.803292 trunc from 0x013be6 to 0x017f2e (0x4349 bytes) 000202 1788644166.815848 trunc from 0x023c12 to 0x03e5cd (0x1a9bc bytes) 000203 1788644166.872555 punch from 0x016d94 to 0x01f3dd (0x864a bytes) 000204 1788644166.878104 read 0x03398f thru 0x03e5cd (0xac3f bytes) 000206 1788644166.894205 MIRROR_RESYNC 2 mirrors 000207 1788644167.623141 write 0x03be87 thru 0x03d5c0 (0x173a bytes) 000208 1788644167.675852 mapread 0x0052ae thru 0x0117c3 (0xc516 bytes) 000209 1788644167.703242 mapread 0x015b74 thru 0x01aea2 (0x532f bytes) 000210 1788644167.725323 trunc from 0x008dfa to 0x00f9aa (0x6bb1 bytes) 000211 1788644167.732284 mapwrite 0x03a5c5 thru 0x03c39c (0x1dd8 bytes) 000212 1788644167.805401 trunc from 0x03b2c8 to 0x03e5cd (0x3306 bytes) 000213 1788644167.816835 mapread 0x0017a2 thru 0x008760 (0x6fbf bytes) 000215 1788644167.845362 read 0x0183c4 thru 0x01bb99 (0x37d6 bytes) 000216 1788644167.846886 mapread 0x01b48b thru 0x026a80 (0xb5f6 bytes) 000217 1788644167.876341 mapwrite 0x000fb7 thru 0x009792 (0x87dc bytes) 000218 1788644167.935072 read 0x02a3a4 thru 0x0346c3 (0xa320 bytes) 000219 1788644167.952196 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000220 1788644168.070206 read 0x022d1a thru 0x028f84 (0x626b bytes) 000221 1788644168.079985 mapread 0x009f7d thru 0x01416c (0xa1f0 bytes) 000222 1788644168.102993 mapwrite 0x0318ef thru 0x03ba7e (0xa190 bytes) 000223 1788644168.138204 mapwrite 0x03f517 thru 0x03ffff (0xae9 bytes) 000224 1788644168.172220 trunc from 0x036d59 to 0x037822 (0xaca bytes) 000225 1788644168.175110 fallocate 0x0285cb thru 0x035227 (0xcc5d bytes) 000226 1788644168.178141 mapwrite 0x00fee3 thru 0x01c427 (0xc545 bytes) 000228 1788644168.193770 trunc from 0x040000 to 0x0399c7 (0x6638 bytes) 000229 1788644168.217997 mapwrite 0x003e13 thru 0x00ef71 (0xb15f bytes) 000230 1788644168.248782 write 0x02a81f thru 0x03249b (0x7c7d bytes) 000231 1788644168.251845 punch from 0x01f93c to 0x021580 (0x1c45 bytes) 000232 1788644168.257063 write 0x02f72b thru 0x03ca01 (0xd2d7 bytes) 000233 1788644168.261111 read 0x02cc04 thru 0x03bd72 (0xf16f bytes) 000234 1788644168.262401 fallocate 0x01966f thru 0x029236 (0xfbc8 bytes) 000235 1788644168.269422 trunc from 0x02688c to 0x030159 (0x98ce bytes) 000236 1788644168.284646 mapwrite 0x0152cd thru 0x01b4e3 (0x6217 bytes) 000237 1788644168.305851 fallocate 0x034e46 thru 0x03c6d3 (0x788e bytes) 000238 1788644168.311767 trunc from 0x03ca02 to 0x021ef9 (0x1ab08 bytes) 000239 1788644168.332415 mapwrite 0x005053 thru 0x009e5e (0x4e0c bytes) 000240 1788644168.360980 punch from 0x01d8ad to 0x021ef9 (0x464d bytes) 000242 1788644168.376428 MIRROR_EXTEND to 2 mirrors 000243 1788644169.374267 mapwrite 0x002772 thru 0x009094 (0x6923 bytes) 000244 1788644169.550762 trunc from 0x01178e to 0x019f83 (0x87f6 bytes) 000245 1788644169.568209 read 0x014cbf thru 0x0186fc (0x3a3e bytes) 000246 1788644169.584323 punch from 0x00f8fe to 0x01bf59 (0xc65c bytes) 000247 1788644169.590384 write 0x02f795 thru 0x035e93 (0x66ff bytes) 000248 1788644169.601142 mapwrite 0x00da78 thru 0x010bf1 (0x317a bytes) 000249 1788644169.636652 fallocate 0x02c9f4 thru 0x0346d1 (0x7cde bytes) 000250 1788644169.643355 read 0x032c61 thru 0x032eeb (0x28b bytes) 000251 1788644169.645322 mapwrite 0x0220af thru 0x025de2 (0x3d34 bytes) 000252 1788644169.690096 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000253 1788644169.861757 write 0x02e740 thru 0x03c6aa (0xdf6b bytes) 000254 1788644169.887251 fallocate 0x03151e thru 0x03d419 (0xbefc bytes) 000255 1788644169.895273 trunc from 0x03d41a to 0x00e9f0 (0x2ea29 bytes) 000256 1788644169.953393 MIRROR_EXTEND to 2 mirrors 000257 1788644171.034940 mapwrite 0x02f94e thru 0x031973 (0x2026 bytes) 000258 1788644171.274854 trunc from 0x031974 to 0x035d39 (0x43c6 bytes) 000259 1788644171.309344 trunc from 0x035d3a to 0x00fb42 (0x261f7 bytes) 000260 1788644171.359092 mapwrite 0x031a89 thru 0x03ffff (0xe577 bytes) 000261 1788644171.482721 mapread 0x00d730 thru 0x013a5d (0x632e bytes) 000262 1788644171.545278 trunc from 0x040000 to 0x00bd41 (0x342be bytes) 000264 1788644171.578187 read 0x002efc thru 0x00bd41 (0x8e46 bytes) 000265 1788644171.590419 mapwrite 0x0167d8 thru 0x021a9d (0xb2c6 bytes) 000266 1788644171.667261 punch from 0x00ad6e to 0x017898 (0xcb2b bytes) 000267 1788644171.675567 read 0x003836 thru 0x00caa1 (0x926c bytes) 000268 1788644171.710372 trunc from 0x0194bc to 0x01b3ab (0x1ef0 bytes) 000269 1788644171.720438 fallocate 0x033ad3 thru 0x03b79e (0x7ccc bytes) 000270 1788644171.740162 trunc from 0x03b79f to 0x00fa3b (0x2bd63 bytes) 000271 1788644171.783117 mapread 0x00c33e thru 0x00fa3b (0x36fe bytes) 000272 1788644171.784156 mapread 0x00efbb thru 0x00fa3b (0xa81 bytes) 000273 1788644171.785032 fallocate 0x008a05 thru 0x008df9 (0x3f5 bytes) 000274 1788644171.791115 mapwrite 0x02aea0 thru 0x02b8e8 (0xa49 bytes) 000275 1788644171.902784 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b8e9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b8e9): Mirror 2: 0x885b110c (stale) CRC-32 checksum value for chunk [0, 0x2b8e9): Mirror 1: 0x5e6bde11 Mirror 3: 0x5e6bde11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b8e9: skipped 000276 1788644173.271920 read 0x001a70 thru 0x00aadf (0x9070 bytes) 000277 1788644173.296314 mapwrite 0x0199d3 thru 0x021a66 (0x8094 bytes) 000278 1788644173.504225 read 0x01c775 thru 0x027dd0 (0xb65c bytes) 000279 1788644173.536320 punch from 0x0148fa to 0x015e8e (0x1595 bytes) 000281 1788644173.543558 punch from 0x028bda to 0x02b8e8 (0x2d0f bytes) 000282 1788644173.552233 fallocate 0x0072ed thru 0x009e77 (0x2b8b bytes) 000283 1788644173.561244 trunc from 0x0114af to 0x02076e (0xf2c0 bytes) 000284 1788644173.569354 read 0x02004e thru 0x020417 (0x3ca bytes) 000285 1788644173.592964 mapread 0x024df4 thru 0x02b8e8 (0x6af5 bytes) 000286 1788644173.594055 mapwrite 0x02db1c thru 0x03c281 (0xe766 bytes) 000287 1788644173.678676 read 0x000d29 thru 0x00d71c (0xc9f4 bytes) 000288 1788644173.679435 trunc from 0x034dbe to 0x03c281 (0x74c4 bytes) 000290 1788644173.686390 fallocate 0x030585 thru 0x0374c6 (0x6f42 bytes) 000291 1788644173.693432 mapwrite 0x038630 thru 0x0386c0 (0x91 bytes) 000292 1788644173.733149 punch from 0x0100bf to 0x01bf5f (0xbea1 bytes) 000293 1788644173.748813 mapread 0x0083b7 thru 0x010050 (0x7c9a bytes) 000294 1788644173.760817 write 0x03dae0 thru 0x03ffff (0x2520 bytes) 000295 1788644173.769067 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x54797bf4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x54797bf4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000296 1788644174.168224 read 0x001c1f thru 0x005f49 (0x432b bytes) 000297 1788644174.182114 mapread 0x00f338 thru 0x018f9a (0x9c63 bytes) 000299 1788644174.197290 write 0x039a23 thru 0x03ffff (0x65dd bytes) 000300 1788644174.209116 MIRROR_RESYNC 2 mirrors 000301 1788644174.873098 read 0x01b218 thru 0x020fcd (0x5db6 bytes) 000302 1788644174.884733 mapread 0x035635 thru 0x035921 (0x2ed bytes) 000304 1788644174.912287 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000305 1788644175.067070 punch from 0x016466 to 0x02266b (0xc206 bytes) 000306 1788644175.112846 MIRROR_EXTEND to 2 mirrors 000307 1788644176.071681 mapread 0x021ddd thru 0x028048 (0x626c bytes) 000308 1788644176.115806 mapwrite 0x03c6a9 thru 0x03e545 (0x1e9d bytes) 000309 1788644176.246427 mapwrite 0x03be79 thru 0x03ffff (0x4187 bytes) 000310 1788644176.296120 MIRROR_RESYNC 2 mirrors 000311 1788644177.651016 trunc from 0x040000 to 0x023838 (0x1c7c7 bytes) 000313 1788644177.736797 mapread 0x00cdd1 thru 0x00ecf6 (0x1f26 bytes) 000314 1788644177.783966 mapwrite 0x00dd71 thru 0x01b385 (0xd615 bytes) 000315 1788644177.855852 mapwrite 0x03c191 thru 0x03c89c (0x70c bytes) 000316 1788644177.974110 mapread 0x01df58 thru 0x02b743 (0xd7ec bytes) 000317 1788644178.004123 fallocate 0x02cf0f thru 0x02eaaa (0x1b9c bytes) 000318 1788644178.010099 trunc from 0x03c89d to 0x01efae (0x1d8ee bytes) 000319 1788644178.037360 trunc from 0x006346 to 0x007894 (0x154f bytes) 000321 1788644178.046011 trunc from 0x003676 to 0x00df3e (0xa8c9 bytes) 000322 1788644178.060758 mapread 0x006e0b thru 0x0106a1 (0x9897 bytes) 000323 1788644178.083109 trunc from 0x01402d to 0x016d17 (0x2ceb bytes) 000324 1788644178.104069 trunc from 0x01efaf to 0x002a0a (0x1c5a4 bytes) 000325 1788644178.139938 mapwrite 0x035a99 thru 0x039d2d (0x4295 bytes) 000326 1788644178.234602 trunc from 0x039d2e to 0x036848 (0x34e5 bytes) 000327 1788644178.259429 punch from 0x01c675 to 0x02a82c (0xe1b8 bytes) 000328 1788644178.277953 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000329 1788644178.510087 punch from 0x01b8d8 to 0x025397 (0x9ac0 bytes) 000330 1788644178.531567 trunc from 0x0167e0 to 0x0216ce (0xaeef bytes) 000331 1788644178.539672 write 0x012851 thru 0x0142ae (0x1a5e bytes) 000332 1788644178.553565 mapread 0x01fe14 thru 0x02c288 (0xc475 bytes) 000333 1788644178.610561 trunc from 0x036849 to 0x005ab2 (0x30d96 bytes) 000334 1788644178.644051 punch from 0x004a89 to 0x005ab2 (0x102a bytes) 000335 1788644178.653225 mapwrite 0x0142a4 thru 0x01c061 (0x7dbe bytes) 000336 1788644178.743146 write 0x005adf thru 0x012b5b (0xd07d bytes) 000337 1788644178.760935 punch from 0x00e32c to 0x01441e (0x60f3 bytes) 000339 1788644178.818116 write 0x01f132 thru 0x02e980 (0xf84f bytes) 000340 1788644178.827117 trunc from 0x02e981 to 0x0205fe (0xe382 bytes) 000342 1788644178.859998 mapread 0x003961 thru 0x00bd99 (0x8439 bytes) 000343 1788644178.873145 mapwrite 0x03a40a thru 0x03ffff (0x5bf6 bytes) 000344 1788644178.960052 mapread 0x00693b thru 0x013be7 (0xd2ad bytes) 000345 1788644178.971991 trunc from 0x040000 to 0x012d09 (0x2d2f6 bytes) 000346 1788644179.000516 write 0x000808 thru 0x01050c (0xfd05 bytes) 000347 1788644179.006365 mapwrite 0x038c20 thru 0x03badb (0x2ebc bytes) 000348 1788644179.138844 MIRROR_EXTEND to 2 mirrors 000349 1788644180.316600 read 0x02fe71 thru 0x031d80 (0x1f10 bytes) 000351 1788644180.340568 write 0x00e744 thru 0x010b89 (0x2446 bytes) 000352 1788644180.437437 mapread 0x02aa8b thru 0x02dec3 (0x3439 bytes) 000353 1788644180.472738 write 0x017fca thru 0x0248b2 (0xc8e9 bytes) 000354 1788644180.507883 punch from 0x0072f1 to 0x0085a2 (0x12b2 bytes) 000356 1788644180.516601 fallocate 0x031efa thru 0x037509 (0x5610 bytes) 000357 1788644180.532676 mapwrite 0x036f3b thru 0x03ffff (0x90c5 bytes) 000359 1788644180.693806 read 0x039c0c thru 0x03ffff (0x63f4 bytes) 000360 1788644180.694436 mapwrite 0x03201c thru 0x033f81 (0x1f66 bytes) 000361 1788644180.728227 fallocate 0x024ac4 thru 0x026166 (0x16a3 bytes) 000362 1788644180.737508 mapread 0x03d6e1 thru 0x03dc0c (0x52c bytes) 000363 1788644180.739244 punch from 0x0399af to 0x03ffff (0x6651 bytes) 000364 1788644180.754037 trunc from 0x02f33f to 0x03e220 (0xeee2 bytes) 000365 1788644180.769044 fallocate 0x005dba thru 0x012d1e (0xcf65 bytes) 000366 1788644180.800648 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000369 1788644181.059455 read 0x019852 thru 0x0264f9 (0xcca8 bytes) 000370 1788644181.082862 trunc from 0x040000 to 0x03702a (0x8fd5 bytes) 000371 1788644181.105778 trunc from 0x0094ad to 0x01706d (0xdbc1 bytes) 000372 1788644181.118283 punch from 0x0007d5 to 0x005f83 (0x57af bytes) 000373 1788644181.123147 mapwrite 0x005f19 thru 0x00f178 (0x9260 bytes) 000375 1788644181.197213 mapwrite 0x02ea32 thru 0x03bdfd (0xd3cc bytes) 000376 1788644181.308110 mapread 0x02349c thru 0x033304 (0xfe69 bytes) 000377 1788644181.309287 mapwrite 0x02ae79 thru 0x035c8a (0xae12 bytes) 000378 1788644181.368132 write 0x01375b thru 0x01517b (0x1a21 bytes) 000379 1788644181.409144 read 0x007d43 thru 0x0157b6 (0xda74 bytes) 000380 1788644181.443828 trunc from 0x01991d to 0x025b71 (0xc255 bytes) 000381 1788644181.485443 read 0x008ae5 thru 0x015289 (0xc7a5 bytes) 000382 1788644181.486073 mapread 0x03125f thru 0x03bdfd (0xab9f bytes) 000383 1788644181.488002 trunc from 0x03bdfe to 0x030ecd (0xaf30 bytes) 000384 1788644181.516115 punch from 0x02f7f7 to 0x030ecd (0x16d7 bytes) 000386 1788644181.532099 trunc from 0x030ece to 0x0103ae (0x20b1f bytes) 000387 1788644181.564099 mapwrite 0x005ef9 thru 0x014941 (0xea49 bytes) 000388 1788644181.632124 MIRROR_EXTEND to 2 mirrors 000389 1788644182.529369 write 0x00e4ab thru 0x010ff1 (0x2b47 bytes) 000391 1788644182.648527 mapwrite 0x025bdd thru 0x02d55d (0x7981 bytes) 000392 1788644182.782684 write 0x02a1e2 thru 0x0353c7 (0xb1e6 bytes) 000394 1788644182.790173 trunc from 0x0353c8 to 0x001ff3 (0x333d4 bytes) 000395 1788644182.826876 punch from 0x000dca to 0x001ff3 (0x122a bytes) 000396 1788644182.835501 mapwrite 0x0273be thru 0x0290bb (0x1cfe bytes) 000397 1788644182.919087 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x290bc): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x290bc): Mirror 2: 0xbf75a1ea (stale) CRC-32 checksum value for chunk [0, 0x290bc): Mirror 1: 0x1b3d763a Mirror 3: 0x1b3d763a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x290bc: skipped 000398 1788644184.255907 trunc from 0x0127fc to 0x021ead (0xf6b2 bytes) 000399 1788644184.342637 mapwrite 0x020d66 thru 0x02c9af (0xbc4a bytes) 000400 1788644184.498943 trunc from 0x02c9b0 to 0x014d33 (0x17c7c bytes) 000401 1788644184.516323 trunc from 0x0054df to 0x00e1a0 (0x8cc2 bytes) 000402 1788644184.523166 read 0x00a01d thru 0x00f153 (0x5137 bytes) 000403 1788644184.541909 punch from 0x002286 to 0x00965b (0x73d6 bytes) 000405 1788644184.548549 mapread 0x00220c thru 0x009b50 (0x7945 bytes) 000406 1788644184.580007 fallocate 0x0245aa thru 0x033793 (0xf1ea bytes) 000407 1788644184.597604 write 0x006d99 thru 0x016b94 (0xfdfc bytes) 000408 1788644184.619814 punch from 0x02d3ae to 0x033793 (0x63e6 bytes) 000409 1788644184.638166 mapwrite 0x01dacf thru 0x026747 (0x8c79 bytes) 000411 1788644184.726127 trunc from 0x033794 to 0x00ca9c (0x26cf7 bytes) 000412 1788644184.790992 trunc from 0x00ca9d to 0x03ff3c (0x334a0 bytes) 000413 1788644184.809560 write 0x0154f3 thru 0x0178cc (0x23da bytes) 000414 1788644184.821017 mapread 0x009c2c thru 0x014c55 (0xb02a bytes) 000415 1788644184.835793 trunc from 0x0323bb to 0x03a396 (0x7fdc bytes) 000416 1788644184.840611 trunc from 0x03d915 to 0x03ff3c (0x2628 bytes) 000419 1788644184.855484 write 0x0171ea thru 0x01be09 (0x4c20 bytes) 000420 1788644184.873753 trunc from 0x03ff3d to 0x017ec5 (0x28077 bytes) 000421 1788644184.904468 trunc from 0x017ec6 to 0x03ca3a (0x24b75 bytes) 000422 1788644184.935521 MIRROR_RESYNC 3 mirrors 000423 1788644186.501436 mapwrite 0x01611d thru 0x01800e (0x1ef2 bytes) 000424 1788644186.679050 trunc from 0x03ca3b to 0x00bcf6 (0x30d44 bytes) 000425 1788644186.751156 fallocate 0x00c868 thru 0x017f40 (0xb6d9 bytes) 000426 1788644186.786622 trunc from 0x0140cc to 0x017f40 (0x3e75 bytes) 000430 1788644186.826212 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x17f41): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17f41): Mirror 2: 0xc82043ed (stale) Mirror 3: 0xc82043ed (stale) CRC-32 checksum value for chunk [0, 0x17f41): Mirror 1: 0xba2f9f05 Mirror 4: 0xba2f9f05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17f41: skipped 000431 1788644187.771572 fallocate 0x01dc8f thru 0x01fe45 (0x21b7 bytes) 000432 1788644187.872056 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x17f41): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17f41): Mirror 2: 0xc82043ed (stale) Mirror 3: 0xc82043ed (stale) Mirror 4: 0xba2f9f05 (stale) CRC-32 checksum value for chunk [0, 0x17f41): Mirror 1: 0xba2f9f05 Mirror 5: 0xba2f9f05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17f41: skipped 000434 1788644189.528832 read 0x00fcfa thru 0x017f40 (0x8247 bytes) 000435 1788644189.561746 trunc from 0x017f41 to 0x033282 (0x1b342 bytes) 000436 1788644189.627896 fallocate 0x03887b thru 0x03ffff (0x7785 bytes) 000437 1788644189.640436 mapwrite 0x03102b thru 0x037306 (0x62dc bytes) 000439 1788644189.712072 trunc from 0x037307 to 0x03b208 (0x3f02 bytes) 000440 1788644189.731447 mapwrite 0x03be34 thru 0x03dacc (0x1c99 bytes) 000441 1788644189.828586 read 0x021432 thru 0x02ab7d (0x974c bytes) 000442 1788644189.842215 trunc from 0x03dacd to 0x0287a3 (0x15329 bytes) 000443 1788644189.872152 fallocate 0x03a1d8 thru 0x03ffff (0x5e28 bytes) 000444 1788644189.878535 trunc from 0x0301d6 to 0x03f28b (0xf0b6 bytes) 000445 1788644189.887006 mapwrite 0x0156d4 thru 0x021ebd (0xc7ea bytes) 000446 1788644189.920559 mapread 0x039fc1 thru 0x03add8 (0xe18 bytes) 000447 1788644189.954071 MIRROR_RESYNC 5 mirrors 000448 1788644191.920747 write 0x00eb7d thru 0x01bcc2 (0xd146 bytes) 000449 1788644191.968994 fallocate 0x02a244 thru 0x0357d8 (0xb595 bytes) 000450 1788644191.974153 write 0x027f03 thru 0x031140 (0x923e bytes) 000451 1788644191.984495 write 0x01cdce thru 0x0249e4 (0x7c17 bytes) 000452 1788644191.994613 punch from 0x02ac22 to 0x0392d9 (0xe6b8 bytes) 000453 1788644192.011608 fallocate 0x004b4b thru 0x0059a4 (0xe5a bytes) 000454 1788644192.017229 mapread 0x01b022 thru 0x02799e (0xc97d bytes) 000456 1788644192.028433 punch from 0x02bcdf to 0x02de47 (0x2169 bytes) 000457 1788644192.032910 write 0x017c50 thru 0x020541 (0x88f2 bytes) 000458 1788644192.034502 trunc from 0x040000 to 0x003a5d (0x3c5a2 bytes) 000459 1788644192.052485 trunc from 0x003a5e to 0x024d4b (0x212ee bytes) 000460 1788644192.073946 punch from 0x0108c6 to 0x014585 (0x3cc0 bytes) 000461 1788644192.079189 fallocate 0x020dbb thru 0x02a4d1 (0x9717 bytes) 000462 1788644192.084056 punch from 0x015776 to 0x016873 (0x10fe bytes) 000463 1788644192.089056 trunc from 0x015eb1 to 0x01d61f (0x776f bytes) 000465 1788644192.093978 read 0x01734b thru 0x02126d (0x9f23 bytes) 000467 1788644192.102985 trunc from 0x004934 to 0x0083d7 (0x3aa4 bytes) 000468 1788644192.107723 mapwrite 0x02f56c thru 0x030187 (0xc1c bytes) 000470 1788644192.148048 punch from 0x013c20 to 0x01d590 (0x9971 bytes) 000471 1788644192.175944 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x30188): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30188): Mirror 2: 0xb981ad3 (stale) Mirror 3: 0xb981ad3 (stale) Mirror 4: 0xb981ad3 (stale) Mirror 5: 0xb981ad3 (stale) CRC-32 checksum value for chunk [0, 0x30188): Mirror 1: 0x88797006 Mirror 6: 0x88797006 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30188: skipped 000472 1788644193.587742 punch from 0x00eb5d to 0x0175b0 (0x8a54 bytes) 000473 1788644193.670883 MIRROR_RESYNC 6 mirrors 000474 1788644195.192901 trunc from 0x030188 to 0x014f1e (0x1b269 bytes) 000475 1788644195.275507 write 0x03407c thru 0x03fd27 (0xbcac bytes) 000476 1788644195.277818 read 0x01c105 thru 0x01c8e3 (0x7df bytes) 000477 1788644195.290919 read 0x00099c thru 0x0105b5 (0xfc1a bytes) 000478 1788644195.311517 read 0x01584c thru 0x01c0a5 (0x685a bytes) 000480 1788644195.320419 fallocate 0x038f14 thru 0x03fc1a (0x6d07 bytes) 000481 1788644195.335333 mapwrite 0x022fe1 thru 0x029176 (0x6196 bytes) 000482 1788644195.388630 punch from 0x01496c to 0x01615e (0x17f3 bytes) 000483 1788644195.395474 read 0x005598 thru 0x014cbb (0xf724 bytes) 000484 1788644195.409727 trunc from 0x016988 to 0x026273 (0xf8ec bytes) 000485 1788644195.422156 trunc from 0x03fd28 to 0x03d9e2 (0x2345 bytes) 000488 1788644195.445953 mapread 0x02576e thru 0x027caa (0x253d bytes) 000491 1788644195.460748 mapwrite 0x01c771 thru 0x021b8d (0x541d bytes) 000492 1788644195.532921 MIRROR_RESYNC 6 mirrors 000493 1788644197.893103 read 0x0023f6 thru 0x008468 (0x6073 bytes) 000494 1788644197.905487 trunc from 0x03d9e3 to 0x023f30 (0x19ab2 bytes) 000495 1788644198.008442 read 0x0216b5 thru 0x023f30 (0x287c bytes) 000498 1788644198.037011 mapwrite 0x010bc2 thru 0x010d5a (0x199 bytes) 000499 1788644198.080567 punch from 0x020cd4 to 0x023f30 (0x325d bytes) 000500 1788644198.106794 fallocate 0x01cccc thru 0x01dba2 (0xed7 bytes) 000501 1788644198.130206 write 0x01bfb0 thru 0x02a9eb (0xea3c bytes) 000503 1788644198.160247 MIRROR_RESYNC 6 mirrors 000504 1788644200.163542 read 0x029b4d thru 0x02a9eb (0xe9f bytes) 000505 1788644200.168943 fallocate 0x00768a thru 0x00fa79 (0x83f0 bytes) 000506 1788644200.218303 fallocate 0x00eee1 thru 0x015fc8 (0x70e8 bytes) 000507 1788644200.224198 trunc from 0x02a9ec to 0x014ce6 (0x15d05 bytes) 000509 1788644200.249567 trunc from 0x014ce7 to 0x008f9e (0xbd48 bytes) 000510 1788644200.272585 write 0x0076a8 thru 0x013a06 (0xc35f bytes) 000511 1788644200.280510 fallocate 0x03eedf thru 0x03ffff (0x1121 bytes) 000512 1788644200.291409 MIRROR_RESYNC 6 mirrors 000513 1788644202.025368 trunc from 0x040000 to 0x01041b (0x2fbe4 bytes) 000514 1788644202.074794 read 0x004b9b thru 0x00f8f0 (0xad56 bytes) 000515 1788644202.085192 trunc from 0x01041c to 0x004fd8 (0xb443 bytes) 000516 1788644202.105440 mapwrite 0x02a735 thru 0x034861 (0xa12d bytes) 000517 1788644202.206152 MIRROR_RESYNC 6 mirrors 000519 1788644203.397468 mapread 0x015ce4 thru 0x016df0 (0x110d bytes) 000520 1788644203.418091 trunc from 0x01b30f to 0x0272af (0xbfa1 bytes) 000521 1788644203.510569 write 0x03ac0d thru 0x03c22f (0x1623 bytes) 000522 1788644203.512406 mapwrite 0x00f2f2 thru 0x010f81 (0x1c90 bytes) 000523 1788644203.604338 trunc from 0x0232c6 to 0x0295c8 (0x6303 bytes) 000524 1788644203.619194 write 0x01915f thru 0x01b3c3 (0x2265 bytes) 000525 1788644203.657592 fallocate 0x021bb2 thru 0x02b5ad (0x99fc bytes) 000526 1788644203.663042 read 0x029a8f thru 0x0304c7 (0x6a39 bytes) 000527 1788644203.678614 write 0x005540 thru 0x006986 (0x1447 bytes) 000528 1788644203.696376 MIRROR_RESYNC 6 mirrors 000529 1788644206.779285 trunc from 0x013732 to 0x0204c6 (0xcd95 bytes) 000530 1788644206.843035 mapwrite 0x027eba thru 0x028b49 (0xc90 bytes) 000531 1788644206.880126 trunc from 0x03c230 to 0x01ac20 (0x2160f bytes) 000532 1788644206.939642 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ac21): Mirror 2: 0x822c1a8 (stale) Mirror 4: 0x822c1a8 (stale) Mirror 5: 0x822c1a8 (stale) Mirror 6: 0x822c1a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ac21): Mirror 2: 0x822c1a8 (stale) Mirror 4: 0x822c1a8 (stale) Mirror 5: 0x822c1a8 (stale) Mirror 6: 0x822c1a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ac21: skipped 000533 1788644207.475304 write 0x039c1e thru 0x03ffff (0x63e2 bytes) 000534 1788644207.503257 mapread 0x03eec8 thru 0x03ffff (0x1138 bytes) 000535 1788644207.505466 write 0x03a0b3 thru 0x03a1fd (0x14b bytes) 000536 1788644207.506650 write 0x02f887 thru 0x02f9dc (0x156 bytes) 000537 1788644207.528118 mapwrite 0x0111f5 thru 0x0121bb (0xfc7 bytes) 000538 1788644207.578740 trunc from 0x040000 to 0x026496 (0x19b69 bytes) 000539 1788644207.599858 mapread 0x025db3 thru 0x026496 (0x6e4 bytes) 000540 1788644207.618437 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26497): Mirror 4: 0x1cf2250a (stale) Mirror 5: 0x1cf2250a (stale) Mirror 6: 0x1cf2250a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26497): Mirror 4: 0x1cf2250a (stale) Mirror 5: 0x1cf2250a (stale) Mirror 6: 0x1cf2250a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26497: skipped 000541 1788644208.031292 read 0x01b99e thru 0x02155f (0x5bc2 bytes) 000543 1788644208.044758 punch from 0x004fe4 to 0x00e3fe (0x941b bytes) 000544 1788644208.049027 fallocate 0x03ee40 thru 0x03ffff (0x11c0 bytes) 000545 1788644208.054115 fallocate 0x02b919 thru 0x03873a (0xce22 bytes) 000546 1788644208.059348 punch from 0x018e21 to 0x01bf3c (0x311c bytes) 000547 1788644208.063897 trunc from 0x040000 to 0x022e7c (0x1d183 bytes) 000549 1788644208.102034 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22e7d): Mirror 4: 0x42d7b441 (stale) Mirror 6: 0x42d7b441 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22e7d): Mirror 4: 0x42d7b441 (stale) Mirror 6: 0x42d7b441 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22e7d: skipped 000550 1788644208.519976 mapread 0x016268 thru 0x022e7c (0xcc15 bytes) 000551 1788644208.553355 fallocate 0x008f72 thru 0x00b932 (0x29c1 bytes) 000552 1788644208.561045 write 0x014de3 thru 0x01ff4d (0xb16b bytes) 000554 1788644208.569381 trunc from 0x022e7d to 0x035ef5 (0x13079 bytes) 000555 1788644208.589421 mapread 0x01bf83 thru 0x01c459 (0x4d7 bytes) 000557 1788644208.592291 mapread 0x02a29d thru 0x033650 (0x93b4 bytes) 000558 1788644208.610825 write 0x037c13 thru 0x03ffff (0x83ed bytes) 000559 1788644208.612368 mapread 0x00643b thru 0x012caf (0xc875 bytes) 000560 1788644208.621386 trunc from 0x040000 to 0x038de0 (0x721f bytes) 000561 1788644208.637801 mapread 0x00fadc thru 0x01c97e (0xcea3 bytes) 000562 1788644208.642741 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38de1): Mirror 6: 0x6114e5c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38de1): Mirror 6: 0x6114e5c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38de1: skipped 000563 1788644208.964340 trunc from 0x034cec to 0x038de0 (0x40f5 bytes) 000564 1788644208.975603 write 0x01a331 thru 0x027cdc (0xd9ac bytes) 000566 1788644209.007580 MIRROR_RESYNC 2 mirrors 000567 1788644210.079310 fallocate 0x00564f thru 0x015043 (0xf9f5 bytes) 000568 1788644210.146554 trunc from 0x028a27 to 0x03566f (0xcc49 bytes) 000569 1788644210.168145 punch from 0x01a154 to 0x01c3a2 (0x224f bytes) 000571 1788644210.182942 write 0x02dc3e thru 0x02fd52 (0x2115 bytes) 000572 1788644210.217203 fallocate 0x011343 thru 0x0119b9 (0x677 bytes) 000573 1788644210.228522 mapread 0x00e599 thru 0x00eeb0 (0x918 bytes) 000574 1788644210.247054 write 0x034854 thru 0x034b72 (0x31f bytes) 000575 1788644210.259853 fallocate 0x03cadb thru 0x03e079 (0x159f bytes) 000576 1788644210.270618 mapread 0x0351b3 thru 0x038de0 (0x3c2e bytes) 000577 1788644210.326856 punch from 0x03569a to 0x038de0 (0x3747 bytes) 000578 1788644210.333334 write 0x00c860 thru 0x00f08c (0x282d bytes) 000579 1788644210.366094 mapread 0x02bad7 thru 0x036a44 (0xaf6e bytes) 000582 1788644210.418374 trunc from 0x0128fe to 0x014feb (0x26ee bytes) 000584 1788644210.453031 MIRROR_RESYNC 2 mirrors 000585 1788644211.584113 mapwrite 0x037374 thru 0x03ffff (0x8c8c bytes) 000586 1788644211.668517 mapread 0x03f413 thru 0x03ffff (0xbed bytes) 000587 1788644211.670389 read 0x02284b thru 0x02522e (0x29e4 bytes) 000588 1788644211.677156 write 0x0011ae thru 0x007d93 (0x6be6 bytes) 000589 1788644211.687143 fallocate 0x02b86d thru 0x035bff (0xa393 bytes) 000590 1788644211.691785 mapread 0x01b6d3 thru 0x01fa53 (0x4381 bytes) 000591 1788644211.706246 MIRROR_RESYNC 2 mirrors 000592 1788644212.843484 trunc from 0x040000 to 0x032bd2 (0xd42d bytes) 000593 1788644212.922172 write 0x0312d7 thru 0x0325ed (0x1317 bytes) 000594 1788644212.936115 trunc from 0x012fdb to 0x015338 (0x235e bytes) 000595 1788644212.943092 read 0x02520b thru 0x032bd2 (0xd9c8 bytes) 000596 1788644212.952793 write 0x001c66 thru 0x00dc31 (0xbfcc bytes) 000597 1788644212.964523 read 0x00f6bd thru 0x0183d6 (0x8d1a bytes) 000598 1788644212.980448 MIRROR_RESYNC 2 mirrors 000599 1788644213.974039 punch from 0x000a76 to 0x00f968 (0xeef3 bytes) 000600 1788644214.041047 trunc from 0x02ea7b to 0x032bd2 (0x4158 bytes) 000601 1788644214.052471 trunc from 0x016161 to 0x02231d (0xc1bd bytes) 000602 1788644214.064591 read 0x0163a0 thru 0x0247ce (0xe42f bytes) 000603 1788644214.085795 read 0x028b8a thru 0x030258 (0x76cf bytes) 000604 1788644214.104212 mapwrite 0x02f4e7 thru 0x035a4f (0x6569 bytes) 000605 1788644214.204790 write 0x01bb3e thru 0x026c2b (0xb0ee bytes) 000606 1788644214.206311 mapwrite 0x02781c thru 0x032715 (0xaefa bytes) 000607 1788644214.298175 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000608 1788644214.505575 read 0x008afa thru 0x0126c6 (0x9bcd bytes) 000609 1788644214.537842 MIRROR_EXTEND to 2 mirrors 000610 1788644215.320316 read 0x032fe9 thru 0x035a4f (0x2a67 bytes) 000611 1788644215.328867 trunc from 0x020161 to 0x025fe0 (0x5e80 bytes) 000612 1788644215.385547 fallocate 0x023098 thru 0x031425 (0xe38e bytes) 000613 1788644215.400041 mapread 0x02f46e thru 0x035a4f (0x65e2 bytes) 000614 1788644215.422525 trunc from 0x01bcb5 to 0x02856e (0xc8ba bytes) 000615 1788644215.428666 trunc from 0x035a50 to 0x019730 (0x1c31f bytes) 000616 1788644215.466931 trunc from 0x019731 to 0x01cb79 (0x3449 bytes) 000617 1788644215.500951 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000618 1788644215.632284 mapread 0x00b4df thru 0x016fae (0xbad0 bytes) 000619 1788644215.661493 MIRROR_EXTEND to 2 mirrors 000621 1788644216.691260 trunc from 0x005657 to 0x00be10 (0x67ba bytes) 000623 1788644216.749903 punch from 0x00f56c to 0x01a74e (0xb1e3 bytes) 000624 1788644216.773537 MIRROR_RESYNC 2 mirrors 000625 1788644217.538752 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000629 1788644217.723392 read 0x000258 thru 0x00763d (0x73e6 bytes) 000630 1788644217.739396 write 0x025d43 thru 0x0356b8 (0xf976 bytes) 000631 1788644217.777886 MIRROR_EXTEND to 2 mirrors 000632 1788644218.845346 trunc from 0x01f567 to 0x0289ea (0x9484 bytes) 000633 1788644218.928922 punch from 0x030a9e to 0x0356b8 (0x4c1b bytes) 000634 1788644218.936528 fallocate 0x01727d thru 0x018f22 (0x1ca6 bytes) 000637 1788644218.945127 trunc from 0x00b1ca to 0x0115a3 (0x63da bytes) 000638 1788644218.950317 trunc from 0x02f6c0 to 0x033503 (0x3e44 bytes) 000639 1788644218.955387 fallocate 0x039af7 thru 0x03ffff (0x6509 bytes) 000640 1788644218.961501 read 0x005b57 thru 0x007024 (0x14ce bytes) 000641 1788644218.976974 MIRROR_RESYNC 2 mirrors 000642 1788644219.870837 write 0x00af4d thru 0x01901f (0xe0d3 bytes) 000643 1788644219.938557 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000644 1788644220.187593 trunc from 0x02f3a7 to 0x0356b8 (0x6312 bytes) 000645 1788644220.195106 fallocate 0x02454c thru 0x02d3d5 (0x8e8a bytes) 000646 1788644220.213364 mapwrite 0x021b6d thru 0x02785d (0x5cf1 bytes) 000647 1788644220.273890 mapread 0x027dd4 thru 0x0317e7 (0x9a14 bytes) 000648 1788644220.298077 mapwrite 0x01f4d0 thru 0x021d2d (0x285e bytes) 000649 1788644220.325851 read 0x02cbb8 thru 0x033d07 (0x7150 bytes) 000650 1788644220.327372 fallocate 0x021e4a thru 0x02da3c (0xbbf3 bytes) 000651 1788644220.335611 trunc from 0x01e15b to 0x02a347 (0xc1ed bytes) 000652 1788644220.342106 punch from 0x01feb4 to 0x029948 (0x9a95 bytes) 000653 1788644220.357193 read 0x02170d thru 0x02acd5 (0x95c9 bytes) 000654 1788644220.370833 fallocate 0x02ca43 thru 0x0398ee (0xceac bytes) 000655 1788644220.375960 read 0x0347a5 thru 0x0398ee (0x514a bytes) 000656 1788644220.383250 fallocate 0x0200d6 thru 0x02a7e4 (0xa70f bytes) 000657 1788644220.390602 mapread 0x001722 thru 0x004138 (0x2a17 bytes) 000658 1788644220.398325 punch from 0x03959b to 0x0398ee (0x354 bytes) 000659 1788644220.406898 read 0x01d3cd thru 0x02a4a8 (0xd0dc bytes) 000660 1788644220.408921 mapread 0x01bf8e thru 0x02bdc3 (0xfe36 bytes) 000661 1788644220.411002 read 0x00e76f thru 0x00ec9b (0x52d bytes) 000662 1788644220.414421 mapread 0x01bfc7 thru 0x02008a (0x40c4 bytes) 000663 1788644220.417336 trunc from 0x0398ef to 0x02154b (0x183a3 bytes) 000664 1788644220.444493 write 0x03cc17 thru 0x03ffff (0x33e9 bytes) 000665 1788644220.447993 trunc from 0x040000 to 0x00cd7f (0x33280 bytes) 000666 1788644220.472258 fallocate 0x036304 thru 0x03fbc5 (0x98c2 bytes) 000667 1788644220.477925 trunc from 0x020792 to 0x02099a (0x209 bytes) 000668 1788644220.483462 write 0x034188 thru 0x034ce8 (0xb61 bytes) 000669 1788644220.495422 mapwrite 0x024720 thru 0x027cfc (0x35dd bytes) 000670 1788644220.510665 write 0x007792 thru 0x01100f (0x987e bytes) 000671 1788644220.513582 punch from 0x0203da to 0x02539d (0x4fc4 bytes) 000672 1788644220.519245 trunc from 0x03fbc6 to 0x01fa0a (0x201bb bytes) 000674 1788644220.538566 write 0x00e214 thru 0x018ea8 (0xac95 bytes) 000675 1788644220.541224 punch from 0x013206 to 0x0169e5 (0x37e0 bytes) 000677 1788644220.556431 punch from 0x00f3ed to 0x014fac (0x5bc0 bytes) 000678 1788644220.562091 trunc from 0x00c0e2 to 0x0160ff (0xa01e bytes) 000679 1788644220.569173 mapwrite 0x03d8e4 thru 0x03e718 (0xe35 bytes) 000680 1788644220.607492 mapread 0x01024b thru 0x01a2f2 (0xa0a8 bytes) 000681 1788644220.639473 fallocate 0x00d6fa thru 0x01b4d8 (0xdddf bytes) 000682 1788644220.644229 fallocate 0x010536 thru 0x01e8fd (0xe3c8 bytes) 000683 1788644220.648793 trunc from 0x03e719 to 0x018b71 (0x25ba7 bytes) 000684 1788644220.669118 punch from 0x007fe1 to 0x010672 (0x8692 bytes) 000685 1788644220.673523 punch from 0x0184b1 to 0x018b71 (0x6c1 bytes) 000686 1788644220.678396 mapwrite 0x01de05 thru 0x01eb54 (0xd50 bytes) 000687 1788644220.713845 fallocate 0x01b7a2 thru 0x02831f (0xcb7e bytes) 000688 1788644220.718821 fallocate 0x0124bb thru 0x0165d1 (0x4117 bytes) 000689 1788644220.722918 trunc from 0x024c15 to 0x02831f (0x370b bytes) 000690 1788644220.731599 mapwrite 0x031c55 thru 0x03a227 (0x85d3 bytes) 000691 1788644220.859876 mapread 0x0258bb thru 0x02ff20 (0xa666 bytes) 000692 1788644220.964969 trunc from 0x012654 to 0x022600 (0xffad bytes) 000693 1788644220.976894 punch from 0x01911b to 0x0204dd (0x73c3 bytes) 000694 1788644220.985381 read 0x013644 thru 0x023024 (0xf9e1 bytes) 000695 1788644221.008884 read 0x025b82 thru 0x033e6a (0xe2e9 bytes) 000696 1788644221.011411 mapread 0x012f21 thru 0x020856 (0xd936 bytes) 000697 1788644221.017339 mapread 0x031847 thru 0x03a227 (0x89e1 bytes) 000698 1788644221.019496 punch from 0x02afb0 to 0x03115a (0x61ab bytes) 000699 1788644221.028660 trunc from 0x024fa6 to 0x028db5 (0x3e10 bytes) 000700 1788644221.040194 punch from 0x02eacd to 0x038c4e (0xa182 bytes) 000701 1788644221.049498 mapread 0x02884e thru 0x02e167 (0x591a bytes) 000702 1788644221.067581 punch from 0x02ff2f to 0x0357e2 (0x58b4 bytes) 000703 1788644221.073983 read 0x001f32 thru 0x0028e8 (0x9b7 bytes) 000704 1788644221.074667 write 0x02fcda thru 0x03b060 (0xb387 bytes) 000705 1788644221.105138 mapwrite 0x036084 thru 0x038033 (0x1fb0 bytes) 000707 1788644221.128274 punch from 0x015f60 to 0x021b6d (0xbc0e bytes) 000708 1788644221.161638 MIRROR_EXTEND to 2 mirrors 000709 1788644222.160975 trunc from 0x03b061 to 0x00a742 (0x3091e bytes) 000710 1788644222.231913 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000712 1788644222.410866 mapwrite 0x0398be thru 0x03adca (0x150d bytes) 000713 1788644222.470213 mapread 0x0031a6 thru 0x00972b (0x6586 bytes) 000714 1788644222.482391 trunc from 0x03adcb to 0x020f17 (0x19eb3 bytes) 000716 1788644222.502881 punch from 0x0070e0 to 0x009dd3 (0x2cf4 bytes) 000718 1788644222.509222 write 0x0206a8 thru 0x023ddf (0x3738 bytes) 000720 1788644222.515878 write 0x01422b thru 0x020813 (0xc5e9 bytes) 000721 1788644222.526412 MIRROR_EXTEND to 2 mirrors 000722 1788644223.557676 mapread 0x01a2a4 thru 0x0213de (0x713b bytes) 000723 1788644223.592774 mapwrite 0x00de55 thru 0x01166c (0x3818 bytes) 000724 1788644223.698206 trunc from 0x015d66 to 0x017697 (0x1932 bytes) 000725 1788644223.704058 write 0x00b70e thru 0x010659 (0x4f4c bytes) 000726 1788644223.716731 punch from 0x016f8d to 0x0228dc (0xb950 bytes) 000727 1788644223.722544 trunc from 0x01c900 to 0x022687 (0x5d88 bytes) 000728 1788644223.727176 mapread 0x017694 thru 0x01da17 (0x6384 bytes) 000730 1788644223.742773 trunc from 0x009cd7 to 0x00a13a (0x464 bytes) 000731 1788644223.750088 punch from 0x014b6c to 0x022e9e (0xe333 bytes) 000732 1788644223.758876 write 0x0253d9 thru 0x02786d (0x2495 bytes) 000733 1788644223.760887 mapread 0x01b511 thru 0x0270eb (0xbbdb bytes) 000735 1788644223.781457 fallocate 0x0234e2 thru 0x02e9b2 (0xb4d1 bytes) 000736 1788644223.787175 punch from 0x01e528 to 0x02b215 (0xccee bytes) 000737 1788644223.815145 read 0x004555 thru 0x010c82 (0xc72e bytes) 000738 1788644223.827001 punch from 0x01702d to 0x025c40 (0xec14 bytes) 000739 1788644223.832348 write 0x01bb9c thru 0x02476c (0x8bd1 bytes) 000740 1788644223.845404 trunc from 0x02e9b3 to 0x03f14e (0x1079c bytes) 000741 1788644223.868398 write 0x03abca thru 0x03ffff (0x5436 bytes) 000742 1788644223.878126 trunc from 0x025333 to 0x026e87 (0x1b55 bytes) 000743 1788644223.888790 fallocate 0x0129c4 thru 0x01cf14 (0xa551 bytes) 000744 1788644223.895757 mapread 0x031533 thru 0x0357fe (0x42cc bytes) 000745 1788644223.937922 MIRROR_RESYNC 2 mirrors 000746 1788644225.116635 trunc from 0x040000 to 0x01b974 (0x2468b bytes) 000747 1788644225.176574 write 0x00afa0 thru 0x01098b (0x59ec bytes) 000748 1788644225.186174 mapwrite 0x0101ff thru 0x01b785 (0xb587 bytes) 000749 1788644225.270498 fallocate 0x000c91 thru 0x00ceed (0xc25d bytes) 000750 1788644225.284889 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1b975): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b975): Mirror 2: 0x5f8ed9c1 (stale) CRC-32 checksum value for chunk [0, 0x1b975): Mirror 1: 0xfac5eaed Mirror 3: 0xfac5eaed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b975: skipped 000751 1788644226.519549 trunc from 0x010574 to 0x01ad95 (0xa822 bytes) 000752 1788644226.585979 punch from 0x017f6d to 0x01b974 (0x3a08 bytes) 000753 1788644226.592901 mapwrite 0x0147e1 thru 0x018f31 (0x4751 bytes) 000754 1788644226.714332 trunc from 0x01b975 to 0x039ab1 (0x1e13d bytes) 000755 1788644226.783819 read 0x0218b3 thru 0x021a43 (0x191 bytes) 000756 1788644226.791635 fallocate 0x016e32 thru 0x01c8dd (0x5aac bytes) 000757 1788644226.797329 mapwrite 0x02a863 thru 0x030a9f (0x623d bytes) 000758 1788644226.825643 fallocate 0x0339ea thru 0x03ffff (0xc616 bytes) 000759 1788644226.830948 write 0x00aae5 thru 0x017ced (0xd209 bytes) 000760 1788644226.841494 punch from 0x01009e to 0x012273 (0x21d6 bytes) 000761 1788644226.857752 read 0x020a72 thru 0x02fda5 (0xf334 bytes) 000762 1788644226.867922 mapwrite 0x01e37b thru 0x02b6f4 (0xd37a bytes) 000763 1788644226.892683 read 0x007a9f thru 0x00d2e4 (0x5846 bytes) 000764 1788644226.893811 mapwrite 0x032fa3 thru 0x03ffff (0xd05d bytes) 000765 1788644226.951937 trunc from 0x040000 to 0x000d18 (0x3f2e7 bytes) 000766 1788644226.979983 trunc from 0x00004c to 0x000d18 (0xccd bytes) 000767 1788644226.992483 MIRROR_RESYNC 3 mirrors 000768 1788644227.586667 write 0x000db2 thru 0x0035b7 (0x2806 bytes) 000769 1788644227.628910 fallocate 0x008e40 thru 0x017683 (0xe844 bytes) 000770 1788644227.635214 trunc from 0x00705b to 0x0090f5 (0x209b bytes) 000771 1788644227.641198 trunc from 0x017684 to 0x0287be (0x1113b bytes) 000772 1788644227.662104 write 0x02a292 thru 0x033d8f (0x9afe bytes) 000773 1788644227.663738 punch from 0x01120a to 0x01b8d7 (0xa6ce bytes) 000774 1788644227.670426 mapwrite 0x027c4e thru 0x03133a (0x96ed bytes) 000775 1788644227.712464 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33d90): Mirror 3: 0x803fd585 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d90): Mirror 3: 0x803fd585 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d90: skipped 000776 1788644228.289990 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33d90): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d90): Mirror 3: 0x803fd585 (stale) CRC-32 checksum value for chunk [0, 0x33d90): Mirror 1: 0xc10ca4d5 Mirror 4: 0xc10ca4d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d90: skipped 000777 1788644229.608583 write 0x021d86 thru 0x02f90a (0xdb85 bytes) 000778 1788644229.705561 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33d90): Mirror 4: 0xc10ca4d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d90): Mirror 4: 0xc10ca4d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d90: skipped 000779 1788644230.122288 trunc from 0x01413c to 0x014c0a (0xacf bytes) 000781 1788644230.129926 fallocate 0x02bd28 thru 0x036b19 (0xadf2 bytes) 000782 1788644230.140512 trunc from 0x033d90 to 0x018a19 (0x1b376 bytes) 000783 1788644230.188545 fallocate 0x01467e thru 0x02370d (0xf090 bytes) 000785 1788644230.197434 write 0x02df23 thru 0x0316fd (0x37db bytes) 000786 1788644230.202440 trunc from 0x018977 to 0x01dc58 (0x52e2 bytes) 000788 1788644230.210272 trunc from 0x0316fe to 0x018e3c (0x188c1 bytes) 000790 1788644230.242610 mapread 0x018b06 thru 0x018e3c (0x337 bytes) 000791 1788644230.248229 read 0x00f2c9 thru 0x0144c5 (0x51fd bytes) 000792 1788644230.267072 read 0x013dc9 thru 0x016f94 (0x31cc bytes) 000793 1788644230.321137 fallocate 0x0000cc thru 0x00c9d8 (0xc90d bytes) 000795 1788644230.329452 trunc from 0x014fd7 to 0x018e3c (0x3e66 bytes) 000796 1788644230.339475 mapread 0x010e11 thru 0x018e3c (0x802c bytes) 000798 1788644230.379391 read 0x008d6b thru 0x009865 (0xafb bytes) 000799 1788644230.380104 trunc from 0x016cfd to 0x018e3c (0x2140 bytes) 000800 1788644230.384959 write 0x01f516 thru 0x020a14 (0x14ff bytes) 000803 1788644230.387325 write 0x03c26f thru 0x03e34a (0x20dc bytes) 000804 1788644230.390831 trunc from 0x03d15d to 0x03e34a (0x11ee bytes) 000805 1788644230.407643 fallocate 0x01eea3 thru 0x024845 (0x59a3 bytes) 000808 1788644230.430013 MIRROR_RESYNC 2 mirrors 000809 1788644231.477779 mapread 0x03a0f3 thru 0x03e34a (0x4258 bytes) 000810 1788644231.510806 MIRROR_EXTEND to 3 mirrors 000811 1788644232.542013 mapwrite 0x010b30 thru 0x02003e (0xf50f bytes) 000812 1788644232.677829 read 0x022afe thru 0x030d73 (0xe276 bytes) 000813 1788644232.700785 punch from 0x032786 to 0x03e34a (0xbbc5 bytes) 000814 1788644232.714780 mapread 0x019363 thru 0x0279f1 (0xe68f bytes) 000815 1788644232.725344 fallocate 0x034ad9 thru 0x036bd3 (0x20fb bytes) 000816 1788644232.750499 mapwrite 0x002ed0 thru 0x00f2c2 (0xc3f3 bytes) 000817 1788644232.835405 mapread 0x034187 thru 0x03df06 (0x9d80 bytes) 000818 1788644232.884824 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e34b): Mirror 4: 0x102395ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e34b): Mirror 4: 0x102395ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e34b: skipped 000819 1788644233.385117 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000821 1788644233.518422 trunc from 0x035140 to 0x03e34a (0x920b bytes) 000822 1788644233.528215 punch from 0x001896 to 0x00ac68 (0x93d3 bytes) 000825 1788644233.537572 fallocate 0x026b7d thru 0x0318c7 (0xad4b bytes) 000826 1788644233.545327 mapwrite 0x031cff thru 0x034d8d (0x308f bytes) 000827 1788644233.624858 punch from 0x001019 to 0x00c6c9 (0xb6b1 bytes) 000828 1788644233.630466 mapread 0x019856 thru 0x023c17 (0xa3c2 bytes) 000829 1788644233.671923 mapwrite 0x038413 thru 0x03ffff (0x7bed bytes) 000830 1788644233.715869 punch from 0x01dfeb to 0x029fa9 (0xbfbf bytes) 000831 1788644233.725003 punch from 0x0234c7 to 0x0279fd (0x4537 bytes) 000832 1788644233.730480 punch from 0x0156af to 0x01d54f (0x7ea1 bytes) 000833 1788644233.736313 write 0x010ae0 thru 0x01d62a (0xcb4b bytes) 000835 1788644233.748811 punch from 0x018150 to 0x025d2b (0xdbdc bytes) 000836 1788644233.777783 mapwrite 0x0165ae thru 0x01ca2f (0x6482 bytes) 000837 1788644233.805530 punch from 0x0216af to 0x02463c (0x2f8e bytes) 000838 1788644233.810363 mapread 0x021185 thru 0x02d092 (0xbf0e bytes) 000839 1788644233.826484 mapwrite 0x00becf thru 0x00e77d (0x28af bytes) 000841 1788644233.859024 mapread 0x0335b5 thru 0x03ffff (0xca4b bytes) 000843 1788644233.863578 trunc from 0x004f42 to 0x00b22a (0x62e9 bytes) 000844 1788644233.871186 read 0x018cf9 thru 0x021c3b (0x8f43 bytes) 000845 1788644233.872649 trunc from 0x025dad to 0x034a92 (0xece6 bytes) 000846 1788644233.878816 write 0x039954 thru 0x03ffff (0x66ac bytes) 000847 1788644233.882227 mapwrite 0x023198 thru 0x031462 (0xe2cb bytes) 000849 1788644233.967889 write 0x010f82 thru 0x0152fa (0x4379 bytes) 000850 1788644233.969003 trunc from 0x040000 to 0x001449 (0x3ebb6 bytes) 000851 1788644233.999907 read 0x000234 thru 0x001449 (0x1216 bytes) 000852 1788644234.006256 read 0x001301 thru 0x001449 (0x149 bytes) 000853 1788644234.006878 punch from 0x00042e to 0x001449 (0x101c bytes) 000854 1788644234.017122 trunc from 0x00144a to 0x0347bb (0x33372 bytes) 000855 1788644234.040130 punch from 0x00ecc0 to 0x01d40b (0xe74c bytes) 000856 1788644234.048187 mapwrite 0x020a30 thru 0x02f31d (0xe8ee bytes) 000857 1788644234.119503 fallocate 0x02c5a6 thru 0x032f32 (0x698d bytes) 000860 1788644234.128396 trunc from 0x0347bc to 0x0224cb (0x122f0 bytes) 000861 1788644234.151281 trunc from 0x0224cc to 0x0051fd (0x1d2ce bytes) 000862 1788644234.173432 punch from 0x002a67 to 0x0051fd (0x2797 bytes) 000863 1788644234.190104 MIRROR_EXTEND to 2 mirrors 000864 1788644234.817315 read 0x0022e7 thru 0x0051fd (0x2f17 bytes) 000865 1788644234.825450 trunc from 0x0051fe to 0x020edf (0x1bce2 bytes) 000866 1788644234.913399 mapwrite 0x028509 thru 0x0316d6 (0x91ce bytes) 000867 1788644234.988017 trunc from 0x0316d7 to 0x01e9da (0x12cfc bytes) 000868 1788644235.009986 punch from 0x01c309 to 0x01e9da (0x26d2 bytes) 000869 1788644235.020118 fallocate 0x00c7cb thru 0x01c5bf (0xfdf5 bytes) 000871 1788644235.029473 mapread 0x0158d5 thru 0x01e726 (0x8e52 bytes) 000872 1788644235.050255 trunc from 0x002235 to 0x00eaba (0xc886 bytes) 000874 1788644235.057388 fallocate 0x01e74b thru 0x02b47e (0xcd34 bytes) 000875 1788644235.068924 MIRROR_RESYNC 2 mirrors 000876 1788644235.878185 read 0x0277ad thru 0x02b47e (0x3cd2 bytes) 000877 1788644235.891139 mapread 0x00f21d thru 0x0193d6 (0xa1ba bytes) 000878 1788644235.920540 mapread 0x0218cb thru 0x027a3d (0x6173 bytes) 000879 1788644235.940932 fallocate 0x03af47 thru 0x03e17a (0x3234 bytes) 000880 1788644235.984202 mapwrite 0x0146b9 thru 0x01e2ec (0x9c34 bytes) 000881 1788644236.021858 trunc from 0x019640 to 0x01d57d (0x3f3e bytes) 000883 1788644236.026777 read 0x00cdc1 thru 0x0180ef (0xb32f bytes) 000885 1788644236.039258 mapwrite 0x03501d thru 0x03794b (0x292f bytes) 000886 1788644236.092550 mapread 0x02619d thru 0x031895 (0xb6f9 bytes) 000887 1788644236.097507 trunc from 0x03794c to 0x012c58 (0x24cf3 bytes) 000888 1788644236.120019 read 0x01147e thru 0x012c58 (0x17db bytes) 000889 1788644236.121603 trunc from 0x008d2e to 0x012c58 (0x9f2b bytes) 000890 1788644236.126444 mapread 0x001b1c thru 0x00b557 (0x9a3c bytes) 000891 1788644236.136685 trunc from 0x012c59 to 0x033955 (0x20cfd bytes) 000892 1788644236.169486 trunc from 0x033956 to 0x02c52a (0x742b bytes) 000893 1788644236.185232 mapwrite 0x02f62b thru 0x0385a4 (0x8f7a bytes) 000894 1788644236.277517 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x385a5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x385a5): Mirror 2: 0xf532d067 (stale) CRC-32 checksum value for chunk [0, 0x385a5): Mirror 1: 0xe289158d Mirror 3: 0xe289158d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x385a5: skipped 000895 1788644237.561783 fallocate 0x028be1 thru 0x02f6e6 (0x6b06 bytes) 000896 1788644237.589820 read 0x02d6cd thru 0x0359b3 (0x82e7 bytes) 000897 1788644237.601681 punch from 0x030db3 to 0x0385a4 (0x77f2 bytes) 000898 1788644237.609125 trunc from 0x0084b7 to 0x00b592 (0x30dc bytes) 000899 1788644237.619443 punch from 0x028a9d to 0x035366 (0xc8ca bytes) 000900 1788644237.641125 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x385a5): Mirror 3: 0xe289158d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x385a5): Mirror 3: 0xe289158d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x385a5: skipped 000901 1788644238.047549 punch from 0x00d586 to 0x013555 (0x5fd0 bytes) 000902 1788644238.057543 mapread 0x036f8e thru 0x0385a4 (0x1617 bytes) 000903 1788644238.088076 punch from 0x020f87 to 0x0288a3 (0x791d bytes) 000905 1788644238.097118 mapread 0x004ac0 thru 0x00b7e6 (0x6d27 bytes) 000906 1788644238.127444 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x385a5): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x385a5): Mirror 3: 0xe289158d (stale) CRC-32 checksum value for chunk [0, 0x385a5): Mirror 1: 0xf532d067 Mirror 4: 0xf532d067 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x385a5: skipped 000907 1788644239.200300 mapwrite 0x006fcf thru 0x0101c2 (0x91f4 bytes) 000908 1788644239.305637 mapread 0x0085c1 thru 0x016192 (0xdbd2 bytes) 000909 1788644239.325244 read 0x010229 thru 0x011ce7 (0x1abf bytes) 000910 1788644239.325985 punch from 0x02485d to 0x028ee8 (0x468c bytes) 000911 1788644239.330726 mapread 0x01a8d1 thru 0x029529 (0xec59 bytes) 000912 1788644239.362334 write 0x01b8ab thru 0x01f71d (0x3e73 bytes) 000913 1788644239.370092 write 0x036e11 thru 0x03ffff (0x91ef bytes) 000914 1788644239.378143 punch from 0x00dd57 to 0x01182f (0x3ad9 bytes) 000915 1788644239.384920 mapwrite 0x03a7e3 thru 0x03ffff (0x581d bytes) 000916 1788644239.402280 write 0x00dc8e thru 0x01af74 (0xd2e7 bytes) 000917 1788644239.408677 mapwrite 0x001068 thru 0x00ffc6 (0xef5f bytes) 000918 1788644239.425426 mapread 0x003b42 thru 0x00e86b (0xad2a bytes) 000919 1788644239.426778 trunc from 0x00473e to 0x009334 (0x4bf7 bytes) 000920 1788644239.431244 trunc from 0x040000 to 0x0276a4 (0x1895b bytes) 000921 1788644239.452623 trunc from 0x01e0fc to 0x01e225 (0x12a bytes) 000922 1788644239.457967 punch from 0x01d4f6 to 0x026651 (0x915c bytes) 000923 1788644239.464925 trunc from 0x0276a5 to 0x02e00c (0x6968 bytes) 000925 1788644239.494862 mapwrite 0x01bfa9 thru 0x028665 (0xc6bd bytes) 000926 1788644239.592345 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e00d): Mirror 4: 0x89c03c01 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e00d): Mirror 4: 0x89c03c01 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e00d: skipped 000927 1788644239.969793 punch from 0x01300f to 0x01fb5e (0xcb50 bytes) 000928 1788644239.983573 fallocate 0x01bdcc thru 0x027e09 (0xc03e bytes) 000929 1788644239.994074 punch from 0x004eef to 0x007116 (0x2228 bytes) 000930 1788644240.002648 fallocate 0x03d0db thru 0x03ffff (0x2f25 bytes) 000931 1788644240.015750 punch from 0x03341b to 0x036128 (0x2d0e bytes) 000932 1788644240.032839 mapwrite 0x0022f5 thru 0x00ab7c (0x8888 bytes) 000933 1788644240.123235 punch from 0x038c45 to 0x03ffff (0x73bb bytes) 000934 1788644240.144873 fallocate 0x027782 thru 0x035eeb (0xe76a bytes) 000935 1788644240.159600 trunc from 0x01db20 to 0x01e873 (0xd54 bytes) 000936 1788644240.167969 mapread 0x022e2b thru 0x031c8d (0xee63 bytes) 000937 1788644240.200189 trunc from 0x040000 to 0x012dce (0x2d231 bytes) 000938 1788644240.229039 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x12dcf): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12dcf): Mirror 4: 0xf65e5937 (stale) CRC-32 checksum value for chunk [0, 0x12dcf): Mirror 1: 0xf3536c57 Mirror 5: 0xf3536c57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12dcf: skipped 000940 1788644241.317011 fallocate 0x01f79f thru 0x025f4f (0x67b1 bytes) 000941 1788644241.415627 write 0x017d03 thru 0x017eca (0x1c8 bytes) 000942 1788644241.426358 mapread 0x00c77c thru 0x017eca (0xb74f bytes) 000943 1788644241.491408 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x17ecb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17ecb): Mirror 4: 0x9c444af2 (stale) Mirror 5: 0xdba34b57 (stale) CRC-32 checksum value for chunk [0, 0x17ecb): Mirror 1: 0xabe5644c Mirror 6: 0xabe5644c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17ecb: skipped 000944 1788644242.973657 punch from 0x00ad67 to 0x013ead (0x9147 bytes) 000945 1788644243.020263 read 0x00a20e thru 0x01173a (0x752d bytes) 000947 1788644243.030339 fallocate 0x03bdca thru 0x03ffff (0x4236 bytes) 000948 1788644243.036913 write 0x00ad80 thru 0x00c833 (0x1ab4 bytes) 000949 1788644243.046885 fallocate 0x035dbd thru 0x03ffff (0xa243 bytes) 000950 1788644243.052142 punch from 0x03becb to 0x03ffff (0x4135 bytes) 000951 1788644243.056749 punch from 0x014b90 to 0x01b94d (0x6dbe bytes) 000952 1788644243.062294 mapwrite 0x0056e4 thru 0x0116a3 (0xbfc0 bytes) 000954 1788644243.110843 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf8d6c66c (stale) Mirror 6: 0xa9d3456a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf8d6c66c (stale) Mirror 6: 0xa9d3456a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000955 1788644243.520579 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa9d3456a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa9d3456a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000956 1788644243.856924 mapwrite 0x01d1c4 thru 0x0266e0 (0x951d bytes) 000957 1788644243.901424 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17c:0x0]: Invalid argument (22) 000959 1788644244.048768 trunc from 0x00a6a4 to 0x016c6f (0xc5cc bytes) 000961 1788644244.062568 write 0x018f68 thru 0x022bac (0x9c45 bytes) 000962 1788644244.095427 punch from 0x01da07 to 0x02bc11 (0xe20b bytes) 000963 1788644244.126084 mapread 0x01cc07 thru 0x026968 (0x9d62 bytes) 000964 1788644244.159688 mapread 0x03848a thru 0x03ffff (0x7b76 bytes) 000965 1788644244.174058 fallocate 0x03100d thru 0x036c68 (0x5c5c bytes) 000966 1788644244.179857 mapwrite 0x038e8f thru 0x03f112 (0x6284 bytes) 000967 1788644244.215647 write 0x034a42 thru 0x03b51a (0x6ad9 bytes) 000968 1788644244.217911 punch from 0x02abee to 0x03101d (0x6430 bytes) 000970 1788644244.240360 mapread 0x00c46a thru 0x01436f (0x7f06 bytes) 000971 1788644244.270639 trunc from 0x040000 to 0x02c7f2 (0x1380d bytes) 000972 1788644244.307208 trunc from 0x015cd0 to 0x01fb51 (0x9e82 bytes) 000973 1788644244.324980 punch from 0x012c4d to 0x018b76 (0x5f2a bytes) 000974 1788644244.330292 write 0x0026b1 thru 0x00ad58 (0x86a8 bytes) 000976 1788644244.352334 mapwrite 0x028d9c thru 0x034ae0 (0xbd45 bytes) 000978 1788644244.520317 fallocate 0x01bf01 thru 0x023394 (0x7494 bytes) 000979 1788644244.525679 write 0x01c826 thru 0x025608 (0x8de3 bytes) 000980 1788644244.539843 mapread 0x00c78e thru 0x01163f (0x4eb2 bytes) 000981 1788644244.541580 trunc from 0x034ae1 to 0x0180d2 (0x1ca0e bytes) 000982 1788644244.587139 punch from 0x007762 to 0x0153bb (0xdc5a bytes) 000983 1788644244.622350 MIRROR_EXTEND to 2 mirrors 000984 1788644245.455563 trunc from 0x0180d3 to 0x01f562 (0x7490 bytes) 000985 1788644245.514355 trunc from 0x000005 to 0x00a2fd (0xa2f9 bytes) 000987 1788644245.524834 mapread 0x011ada thru 0x01389b (0x1dc2 bytes) 000988 1788644245.555481 MIRROR_RESYNC 2 mirrors 000989 1788644246.103144 mapwrite 0x035619 thru 0x03d0f5 (0x7add bytes) 000990 1788644246.234351 mapread 0x03977b thru 0x03a038 (0x8be bytes) 000991 1788644246.235679 write 0x0272ac thru 0x02b008 (0x3d5d bytes) 000993 1788644246.251271 fallocate 0x039f3b thru 0x03ffff (0x60c5 bytes) 000994 1788644246.263342 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d0f6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0f6): Mirror 2: 0xe46230b6 (stale) CRC-32 checksum value for chunk [0, 0x3d0f6): Mirror 1: 0x1a55f115 Mirror 3: 0x1a55f115 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0f6: skipped 000995 1788644247.439012 punch from 0x031e9a to 0x039e24 (0x7f8b bytes) 000996 1788644247.556035 punch from 0x03454c to 0x035a5b (0x1510 bytes) 000997 1788644247.582016 trunc from 0x0236fb to 0x030acd (0xd3d3 bytes) 000998 1788644247.613447 mapwrite 0x002990 thru 0x00e069 (0xb6da bytes) 000999 1788644247.747005 mapwrite 0x01bca1 thru 0x01d682 (0x19e2 bytes) 001000 1788644247.820427 punch from 0x0207bf to 0x025f59 (0x579b bytes) All operations completed A-OK! PASS 70a (110s) == sanity-flr test 71: check mirror extend parent fid ==== 17:37:34 (1788644254) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1be:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (14s) == sanity-flr test 72: check append on FLR file ========== 17:37:49 (1788644269) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.30006 s, 7.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671831 s, 15.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 17:37:58 (1788644278) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5371360 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1460320 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3180480 .. Extending file size to 6135520 .. failed Extending file size to 7047808 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6890144 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8015072 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7537472 .. Extending file size to 8602976 .. Extending file size to 8629824 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9332768 .. Extending file size to 8833056 .. Waiting 61298 61299 61301 61303 61305 done 192.168.203.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) 192.168.203.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (89s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:39:27 (1788644367) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6319648 .. Extending file size to 722080 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8207840 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8345344 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8897440 .. Extending file size to 9033344 .. Extending file size to 9199488 .. Extending file size to 8912288 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9075552 .. Extending file size to 9188640 .. Extending file size to 9346656 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62940 62941 62943 62945 62948 done 192.168.203.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) 192.168.203.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:40:52 (1788644452) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2148 1269156 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18052 3588968 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26080 3580940 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44132 7169908 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2148 1269156 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18052 3588968 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 26080 3580940 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44132 7169908 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126066 s, 16.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.225666 s, 9.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa88eb470 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:41:13 (1788644473) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120016 s, 17.5 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 17:41:21 (1788644481) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:41:27 (1788644487) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.099709 s, 21.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:41:36 (1788644496) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131398 s, 16.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:41:44 (1788644504) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104166 s, 20.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:42:19 (1788644539) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg322-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111119 s, 9.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x210:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x211:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x210:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x210:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x211:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x210:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x211:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:43:12 (1788644592) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg322-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123005 s, 17.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:44:04 (1788644644) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg322-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg322-server: Pool lustre.test_204f created Adding targets to pool oleg322-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg322-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0836675 s, 25.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x212:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x212:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x212:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x212:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg322-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg322-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg322-server: Pool lustre.test_204f destroyed PASS 204f (70s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:45:14 (1788644714) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:45:20 (1788644720) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:45:28 (1788644728) oleg322-server: Pool lustre.test_206 created Adding targets to pool oleg322-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg322-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg322-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg322-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg322-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:45:56 (1788644756) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732446 s, 14.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500015 s, 21.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4222f9b5 Mirror 3: 0x4222f9b5 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:46:07 (1788644767) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:46:12 (1788644772) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:46:16 (1788644776) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0410612 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1660000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2380000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2380000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2380000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x25e0000 Read this (4 bytes): BBBB 192.168.203.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:47:00 (1788644820) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0110858 s, 739 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:47:07 (1788644827) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567442 s, 18.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 17:47:09.000000000 -0400 Modify: 2026-09-05 17:47:09.000000000 -0400 Change: 2026-09-05 17:47:11.000000000 -0400 Birth: 2026-09-05 17:47:08.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:47:16 (1788644836) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545165 s, 19.2 MB/s oleg322-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg322-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg322-server: O/280000401/d28/15c: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:47:39 (1788644859) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.225915 s, 181 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0203089 s, 202 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 5 17:47 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x84:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 5 17:47 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:47:46 (1788644866) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:47:54 (1788644874) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1923 sec =========== 17:48:03 (1788644883) === sanity-flr: start cleanup 17:48:04 (1788644884) === === sanity-flr: finish cleanup 17:48:13 (1788644893) ===