-----============= acceptance-small: sanity-flr ============----- Sun Jun 7 00:51:42 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg209-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 00:51:51 (1780807911) === oleg209-client.virtnet: executing check_config_client /mnt/lustre oleg209-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg209-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff92b9c974f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff92b9c974f800.idle_timeout=debug disable quota as required oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 00:52:05 (1780807925) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8360] == sanity-flr test 0a: lfs mirror create with -N option == 00:52:06 (1780807926) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 00:52:14 (1780807934) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 00:52:16 (1780807936) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 00:52:18 (1780807938) PASS 0d (12s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 00:52:30 (1780807950) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 00:52:32 (1780807952) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 00:52:34 (1780807954) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff92b9c974f800.stats=clear osc.lustre-OST0001-osc-ffff92b9c974f800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 00:52:38 (1780807958) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 00:52:43 (1780807963) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 00:52:48 (1780807968) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 00:52:49 (1780807969) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 00:52:54 (1780807974) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 00:52:59 (1780807979) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 00:53:02 (1780807982) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 00:53:06 (1780807986) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 00:53:10 (1780807990) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 00:53:15 (1780807995) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 00:53:19 (1780807999) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 1.07535 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557627 s, 18.8 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 00:53:24 (1780808004) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0331033 s, 95.0 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 00:53:28 (1780808008) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 00:53:32 (1780808012) Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92b9c974f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92b9c974f800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92b9c974f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92b9c974f800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg209-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (31s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 00:54:03 (1780808043) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0953432 s, 44.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92b9c974f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92b9c974f800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (17s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 00:54:21 (1780808061) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92b9c974f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92b9c974f800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92b9c974f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92b9c974f800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg209-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (37s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 00:54:58 (1780808098) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.07265 s, 17.1 MB/s -rw-r--r-- 1 root root 52428800 Jun 7 00:55 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.633s user 0m0.002s sys 0m0.143s Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92b9c974f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92b9c974f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.518s user 0m0.001s sys 0m0.169s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92b9c974f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92b9c974f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.573s user 0m0.000s sys 0m0.145s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg209-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92b9c974f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92b9c974f800.ost_server_uuid in FULL state after 0 sec PASS 33b (44s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 00:55:42 (1780808142) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 00:55:43 (1780808143) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 00:55:44 (1780808144) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 00:55:46 (1780808146) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 00:55:49 (1780808149) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.223271 s, 56.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 00:55:55 (1780808155) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261037 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014618 s, 71.7 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 00:55:58 (1780808158) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 00:56:02 (1780808162) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000899262 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000936129 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00085997 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000741177 s, 1.4 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 00:56:06 (1780808166) create test files with size 15752552 .. 77acbdab2c23afe16511325e84a520df - 049b22863f5126e5f17f5f5176aa8b68 - 6c99fd3ba7f0f25b5440de55160abf8a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 00:56:14 (1780808174) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 00:56:20 (1780808180) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 00:56:22 (1780808182) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589369 s, 17.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131119 s, 80.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 00:56:26 (1780808186) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0768725 s, 27.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.15821 s, 26.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0715056 s, 29.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (7s) == sanity-flr test 42: lfs mirror verify ================= 00:56:33 (1780808193) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 00:56:34 (1780808194) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 00:56:35 (1780808195) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153859 s, 68.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 00:56:38 (1780808198) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.112738 s, 27.9 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 00:56:43 (1780808203) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 00:56:46 (1780808206) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.147648 s, 71.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 00:57:01 (1780808221) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff92b9d98fd000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.113756 s, 92.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff92b9ca440800.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 00:57:07 (1780808227) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 00:57:22 (1780808242) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (11s) == sanity-flr test 47: Verify mirror obj alloc =========== 00:57:33 (1780808253) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 00:57:34 (1780808254) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0884935 s, 35.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.056598 s, 55.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf5af551 (nosync,stale) Mirror 2: 0xf5af551 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 1e150a50656aa1902c8f4c4839fa061d - mirror 1 checksum: f6c1dcaad6984fceb8107489cb190b6b - mirror 2 checksum: 1e150a50656aa1902c8f4c4839fa061d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Jun 7 00:57:35 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Jun 7 00:57:35 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1e150a50656aa1902c8f4c4839fa061d - mirror 1 checksum: f6c1dcaad6984fceb8107489cb190b6b - mirror 2 checksum: f6c1dcaad6984fceb8107489cb190b6b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Jun 7 00:57:35 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Jun 7 00:57:35 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 00:57:38 (1780808258) Failing mds1 on oleg209-server Stopping /mnt/lustre-mds1 (opts:) on oleg209-server 00:57:40 (1780808260) shut down facet: mds1 facet_host: oleg209-server facet_failover_host: oleg209-server Failover mds1 to oleg209-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 00:57:55 (1780808275) targets are mounted 00:57:55 (1780808275) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xe:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145064 s, 72.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xe lmm_fid: [0x200000405:0xe:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 50A (26s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 00:58:04 (1780808284) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 00:58:05 (1780808285) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 00:58:06 (1780808286) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00791066 s, 2.1 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 00:58:10 (1780808290) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 00:58:12 (1780808292) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 00:58:14 (1780808294) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) PASS 60b (11s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 00:58:24 (1780808304) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 00:58:45 (1780808325) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 00:59:04 (1780808344) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 00:59:23 (1780808363) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 00:59:26 (1780808366) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 55493 mirror split pid 55494 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 55493 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 55494 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 01:00:29 (1780808429) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 8430 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000005 1780808430.146189 mapwrite 0x0055c2 thru 0x0121dd (0xcc1c bytes) 000006 1780808430.183560 read 0x003c15 thru 0x00f96d (0xbd59 bytes) 000010 1780808430.185958 write 0x01680c thru 0x021e33 (0xb628 bytes) 000013 1780808430.188878 mapwrite 0x023535 thru 0x02a3b5 (0x6e81 bytes) truncating to largest ever: 0xc830 000020 1780808430.220838 trunc from 0x02a3b6 to 0x00c82f (0x1db86 bytes) 000025 1780808430.236072 MIRROR_EXTEND to 2 mirrors 000026 1780808430.480637 mapread 0x008fdb thru 0x00c82f (0x3855 bytes) 000029 1780808430.491253 mapwrite 0x00e263 thru 0x00e774 (0x512 bytes) 000030 1780808430.535916 read 0x00a033 thru 0x00e774 (0x4742 bytes) 000031 1780808430.538085 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xe775): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe775): Mirror 2: 0x6c12586a (stale) CRC-32 checksum value for chunk [0, 0xe775): Mirror 1: 0x29191d7a Mirror 3: 0x29191d7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe775: skipped 000032 1780808430.814334 read 0x003697 thru 0x00e774 (0xb0de bytes) 000035 1780808430.821519 mapwrite 0x00420f thru 0x01176a (0xd55c bytes) 000036 1780808430.924610 mapwrite 0x011827 thru 0x019ebc (0x8696 bytes) 000039 1780808430.967984 mapread 0x00ef13 thru 0x0133f8 (0x44e6 bytes) 000040 1780808430.969125 mapwrite 0x02a152 thru 0x0313b6 (0x7265 bytes) 000043 1780808431.013998 MIRROR_RESYNC 3 mirrors 000045 1780808431.327683 write 0x0295bb thru 0x02f89b (0x62e1 bytes) 000046 1780808431.350986 read 0x01d276 thru 0x02aadc (0xd867 bytes) 000049 1780808431.360221 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x313b7): Mirror 3: 0x9f8af6a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x313b7): Mirror 3: 0x9f8af6a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x313b7: skipped 000050 1780808431.532847 read 0x01ae83 thru 0x01fa3f (0x4bbd bytes) 000051 1780808431.537584 read 0x015186 thru 0x01fa5a (0xa8d5 bytes) truncating to largest ever: 0x33ebe 000058 1780808431.546012 trunc from 0x0313b7 to 0x033ebd (0x2b07 bytes) 000059 1780808431.554383 write 0x01993a thru 0x02136b (0x7a32 bytes) 000060 1780808431.555716 write 0x00b897 thru 0x01493a (0x90a4 bytes) 000064 1780808431.558558 read 0x0179ed thru 0x01fc6f (0x8283 bytes) 000065 1780808431.559244 mapwrite 0x00691b thru 0x00ea5a (0x8140 bytes) 000066 1780808431.598610 MIRROR_RESYNC 2 mirrors 000067 1780808431.841328 trunc from 0x033ebe to 0x005165 (0x2ed58 bytes) 000068 1780808431.884588 write 0x03620e thru 0x03d8f1 (0x76e4 bytes) 000069 1780808431.888253 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d8f2): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d8f2): Mirror 3: 0xb48c2dce (stale) CRC-32 checksum value for chunk [0, 0x3d8f2): Mirror 1: 0x20056bdc Mirror 4: 0x20056bdc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d8f2: skipped 000070 1780808432.272249 trunc from 0x03d8f2 to 0x030ad0 (0xce21 bytes) 000071 1780808432.300240 read 0x00df5a thru 0x00fa12 (0x1ab9 bytes) 000073 1780808432.305228 mapread 0x007b5d thru 0x015ef6 (0xe39a bytes) 000074 1780808432.330821 read 0x002617 thru 0x007811 (0x51fb bytes) 000075 1780808432.331561 trunc from 0x030ad1 to 0x0303ee (0x6e2 bytes) 000077 1780808432.348930 trunc from 0x0303ef to 0x01d58e (0x12e60 bytes) 000078 1780808432.356500 write 0x00bd7d thru 0x01b22b (0xf4af bytes) 000079 1780808432.363634 MIRROR_RESYNC 3 mirrors 000080 1780808432.710554 mapwrite 0x0011c7 thru 0x007023 (0x5e5d bytes) 000081 1780808432.807865 read 0x01d444 thru 0x01d58e (0x14b bytes) 000083 1780808432.811585 mapwrite 0x00a4bb thru 0x00e4bb (0x4001 bytes) 000084 1780808432.841137 read 0x01bb1c thru 0x01ced8 (0x13bd bytes) 000085 1780808432.847894 mapwrite 0x00a823 thru 0x010f7e (0x675c bytes) 000087 1780808432.867013 mapwrite 0x009bb4 thru 0x018bb6 (0xf003 bytes) 000088 1780808432.887128 mapread 0x0062c5 thru 0x0113e6 (0xb122 bytes) truncating to largest ever: 0x34d63 000090 1780808432.889043 trunc from 0x01d58f to 0x034d62 (0x177d4 bytes) 000092 1780808432.907523 mapwrite 0x02529c thru 0x02b698 (0x63fd bytes) 000093 1780808432.938479 read 0x025a9f thru 0x02dcc6 (0x8228 bytes) 000094 1780808432.944994 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34d63): Mirror 3: 0x67872cdb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d63): Mirror 3: 0x67872cdb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d63: skipped truncating to largest ever: 0x3aba2 000096 1780808433.081769 trunc from 0x034d63 to 0x03aba1 (0x5e3f bytes) 000097 1780808433.088656 mapwrite 0x00a574 thru 0x00bc5f (0x16ec bytes) 000098 1780808433.112885 mapwrite 0x00642d thru 0x011e96 (0xba6a bytes) 000099 1780808433.141882 write 0x02cc18 thru 0x02d1f3 (0x5dc bytes) 000101 1780808433.147793 trunc from 0x03aba2 to 0x018347 (0x2285a bytes) 000102 1780808433.154924 read 0x014607 thru 0x018347 (0x3d41 bytes) 000103 1780808433.155603 mapread 0x01390c thru 0x018347 (0x4a3c bytes) 000104 1780808433.158946 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000106 1780808433.241901 mapwrite 0x02729c thru 0x035c98 (0xe9fd bytes) 000109 1780808433.305105 write 0x020142 thru 0x02d3e2 (0xd2a1 bytes) 000113 1780808433.317086 mapread 0x01d508 thru 0x023793 (0x628c bytes) 000114 1780808433.318571 write 0x029838 thru 0x02e092 (0x485b bytes) 000117 1780808433.321576 write 0x027356 thru 0x03533b (0xdfe6 bytes) 000118 1780808433.322594 trunc from 0x035c99 to 0x0309da (0x52be bytes) 000119 1780808433.351043 MIRROR_EXTEND to 2 mirrors 000121 1780808433.679169 trunc from 0x0309db to 0x006f66 (0x29a74 bytes) 000122 1780808433.711505 trunc from 0x006f67 to 0x03653b (0x2f5d5 bytes) 000123 1780808433.719526 write 0x00e2e2 thru 0x01642c (0x814b bytes) 000124 1780808433.724169 read 0x00fc7f thru 0x01c41a (0xc79c bytes) 000126 1780808433.729197 write 0x02035a thru 0x02983a (0x94e1 bytes) 000128 1780808433.737549 read 0x01f467 thru 0x028a1c (0x95b6 bytes) 000129 1780808433.741651 MIRROR_RESYNC 2 mirrors 000130 1780808434.038538 MIRROR_EXTEND to 3 mirrors 000131 1780808434.407155 write 0x03305f thru 0x03494a (0x18ec bytes) 000133 1780808434.429413 write 0x009c9c thru 0x00c772 (0x2ad7 bytes) 000135 1780808434.433452 trunc from 0x03653c to 0x000b07 (0x35a34 bytes) 000136 1780808434.441715 trunc from 0x000b08 to 0x0200fa (0x1f5f3 bytes) 000137 1780808434.450003 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x200fb): Mirror 3: 0x92287e1a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x200fb): Mirror 3: 0x92287e1a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x200fb: skipped 000139 1780808434.555978 mapread 0x00bd9b thru 0x017b58 (0xbdbe bytes) 000141 1780808434.569984 mapread 0x01c0ff thru 0x0200fa (0x3ffc bytes) 000142 1780808434.574080 mapwrite 0x013923 thru 0x013ae2 (0x1c0 bytes) 000143 1780808434.588165 read 0x00caf8 thru 0x011147 (0x4650 bytes) 000145 1780808434.591128 MIRROR_RESYNC 2 mirrors 000146 1780808434.818028 trunc from 0x0200fb to 0x013320 (0xcdda bytes) 000147 1780808434.859146 read 0x00bece thru 0x013320 (0x7453 bytes) 000148 1780808434.863376 write 0x038a15 thru 0x03d7ba (0x4da6 bytes) 000149 1780808434.864716 trunc from 0x03d7bb to 0x01640a (0x273b0 bytes) 000150 1780808434.877722 mapwrite 0x01b1dc thru 0x024bf0 (0x9a15 bytes) 000152 1780808434.938761 write 0x01342f thru 0x01f3ba (0xbf8c bytes) 000153 1780808434.940219 mapread 0x0102c4 thru 0x01d945 (0xd682 bytes) 000154 1780808434.941639 trunc from 0x024bf1 to 0x010df0 (0x13e00 bytes) 000155 1780808434.954522 write 0x03b13d thru 0x03ffff (0x4ec3 bytes) 000156 1780808434.955939 write 0x03c45d thru 0x03e383 (0x1f27 bytes) 000158 1780808434.957507 read 0x006646 thru 0x007360 (0xd1b bytes) 000159 1780808434.968001 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000160 1780808435.020704 write 0x024aac thru 0x030187 (0xb6dc bytes) 000166 1780808435.026850 mapwrite 0x0299b1 thru 0x035a92 (0xc0e2 bytes) 000167 1780808435.058139 mapwrite 0x034982 thru 0x03f19b (0xa81a bytes) 000171 1780808435.091081 MIRROR_EXTEND to 2 mirrors 000173 1780808435.421690 write 0x0065bc thru 0x012f73 (0xc9b8 bytes) 000176 1780808435.460966 MIRROR_RESYNC 2 mirrors 000177 1780808435.756763 MIRROR_RESYNC 2 mirrors 000179 1780808435.944966 mapread 0x02a7fa thru 0x02ff47 (0x574e bytes) 000181 1780808435.957372 write 0x01b0ff thru 0x01c952 (0x1854 bytes) 000182 1780808435.990174 mapwrite 0x00939d thru 0x00ca0b (0x366f bytes) 000184 1780808436.028959 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000185 1780808436.092828 write 0x03dd06 thru 0x03ffff (0x22fa bytes) 000187 1780808436.096695 mapwrite 0x02b422 thru 0x039994 (0xe573 bytes) 000189 1780808436.127584 trunc from 0x040000 to 0x0283c0 (0x17c3f bytes) 000191 1780808436.140182 read 0x0252d1 thru 0x0283c0 (0x30f0 bytes) 000192 1780808436.140869 mapread 0x0134ad thru 0x020df7 (0xd94b bytes) 000193 1780808436.161736 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3b05a 000194 1780808436.502630 trunc from 0x0283c1 to 0x03b059 (0x12c99 bytes) 000199 1780808436.569855 mapwrite 0x03d62d thru 0x03ffff (0x29d3 bytes) 000201 1780808436.626202 write 0x02c62b thru 0x031923 (0x52f9 bytes) 000202 1780808436.635875 trunc from 0x040000 to 0x024975 (0x1b68a bytes) 000204 1780808436.645451 mapread 0x0091cd thru 0x009df1 (0xc25 bytes) 000205 1780808436.651758 trunc from 0x024976 to 0x0359d3 (0x1105e bytes) 000208 1780808436.664236 trunc from 0x0359d4 to 0x02e392 (0x7641 bytes) 000209 1780808436.673441 mapread 0x01db6f thru 0x0281cb (0xa65d bytes) 000212 1780808436.685256 mapwrite 0x009e93 thru 0x00dfd1 (0x413f bytes) 000215 1780808436.724300 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e393): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e393): Mirror 2: 0x84f77d67 (stale) CRC-32 checksum value for chunk [0, 0x2e393): Mirror 1: 0xe8bf0faf Mirror 3: 0xe8bf0faf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e393: skipped 000218 1780808437.171080 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e393): Mirror 2: 0x84f77d67 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e393): Mirror 2: 0x84f77d67 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e393: skipped 000219 1780808437.299034 mapread 0x01562f thru 0x01b07f (0x5a51 bytes) 000221 1780808437.309852 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e393): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e393): Mirror 2: 0x84f77d67 (stale) CRC-32 checksum value for chunk [0, 0x2e393): Mirror 1: 0xe8bf0faf Mirror 3: 0xe8bf0faf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e393: skipped 000222 1780808437.679722 mapwrite 0x00e503 thru 0x014e7e (0x697c bytes) 000223 1780808437.800564 mapread 0x007383 thru 0x012595 (0xb213 bytes) 000224 1780808437.814646 write 0x015cea thru 0x01965b (0x3972 bytes) 000225 1780808437.828940 write 0x02b3ea thru 0x02ee1e (0x3a35 bytes) 000229 1780808437.845524 read 0x00f3b3 thru 0x010ac5 (0x1713 bytes) 000230 1780808437.846284 trunc from 0x02ee1f to 0x024ec6 (0x9f58 bytes) 000231 1780808437.861951 write 0x01aeae thru 0x029c9d (0xedf0 bytes) 000232 1780808437.871859 read 0x016407 thru 0x01cdcb (0x69c5 bytes) 000233 1780808437.872580 mapread 0x00461f thru 0x00b4f8 (0x6eda bytes) 000235 1780808437.874614 mapread 0x00e34f thru 0x00fe0d (0x1abf bytes) 000236 1780808437.875683 mapread 0x00a766 thru 0x00c042 (0x18dd bytes) 000237 1780808437.876765 mapread 0x0105b7 thru 0x011edb (0x1925 bytes) 000238 1780808437.877977 mapwrite 0x0044a0 thru 0x00afaf (0x6b10 bytes) 000241 1780808437.917691 mapread 0x003eff thru 0x00ea50 (0xab52 bytes) 000242 1780808437.923768 MIRROR_RESYNC 3 mirrors 000243 1780808438.362136 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000244 1780808438.538074 trunc from 0x029c9e to 0x00c289 (0x1da14 bytes) 000246 1780808438.579058 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000248 1780808438.654261 trunc from 0x00c28a to 0x00ec1f (0x2996 bytes) 000250 1780808438.665602 MIRROR_EXTEND to 2 mirrors 000252 1780808438.959374 write 0x036105 thru 0x03ffff (0x9efb bytes) 000254 1780808439.005640 trunc from 0x040000 to 0x03457a (0xba85 bytes) 000259 1780808439.020398 write 0x03c9e1 thru 0x03ffff (0x361f bytes) 000264 1780808439.024943 read 0x00d0bf thru 0x015cc3 (0x8c05 bytes) 000266 1780808439.031533 mapread 0x03889f thru 0x03be9a (0x35fc bytes) 000267 1780808439.044213 mapwrite 0x02ee21 thru 0x0351b0 (0x6390 bytes) 000271 1780808439.086079 mapwrite 0x0191b9 thru 0x01f3e7 (0x622f bytes) 000272 1780808439.127877 read 0x001903 thru 0x006f2c (0x562a bytes) 000274 1780808439.133889 mapread 0x036724 thru 0x03ffff (0x98dc bytes) 000275 1780808439.135449 mapread 0x010395 thru 0x0109a0 (0x60c bytes) 000277 1780808439.137013 read 0x014d67 thru 0x01d81a (0x8ab4 bytes) 000281 1780808439.143029 MIRROR_RESYNC 2 mirrors 000282 1780808439.560111 read 0x003f84 thru 0x00e3bf (0xa43c bytes) 000284 1780808439.565892 mapread 0x03463f thru 0x0358d9 (0x129b bytes) 000286 1780808439.572208 mapwrite 0x015c10 thru 0x01bb66 (0x5f57 bytes) 000287 1780808439.704400 read 0x03caee thru 0x03ffff (0x3512 bytes) 000289 1780808439.724821 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb12092bc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x20d17a0b Mirror 3: 0x20d17a0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000292 1780808440.128121 write 0x02c7ad thru 0x032b8b (0x63df bytes) 000293 1780808440.222536 mapread 0x016224 thru 0x0222ea (0xc0c7 bytes) 000297 1780808440.249047 read 0x034639 thru 0x03c218 (0x7be0 bytes) 000298 1780808440.275272 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x20d17a0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x20d17a0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000299 1780808440.457338 trunc from 0x040000 to 0x013881 (0x2c77e bytes) 000303 1780808440.485165 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000309 1780808440.597976 write 0x02feee thru 0x037f74 (0x8087 bytes) 000310 1780808440.599903 mapwrite 0x031098 thru 0x03ffff (0xef68 bytes) 000313 1780808440.681033 read 0x03707e thru 0x03ffff (0x8f82 bytes) 000314 1780808440.681529 write 0x006104 thru 0x00f1f0 (0x90ed bytes) 000316 1780808440.694067 read 0x00daf9 thru 0x01accf (0xd1d7 bytes) 000318 1780808440.703356 write 0x038a8c thru 0x03fc84 (0x71f9 bytes) 000320 1780808440.705174 mapwrite 0x014dd0 thru 0x01a94f (0x5b80 bytes) 000321 1780808440.754218 mapread 0x02e536 thru 0x0328fe (0x43c9 bytes) 000322 1780808440.758363 mapread 0x026076 thru 0x030711 (0xa69c bytes) 000323 1780808440.760651 write 0x0397d6 thru 0x03ffff (0x682a bytes) 000327 1780808440.764294 write 0x005384 thru 0x00c5ad (0x722a bytes) 000329 1780808440.766365 trunc from 0x040000 to 0x02a9e4 (0x1561b bytes) 000330 1780808440.779861 trunc from 0x02a9e5 to 0x033632 (0x8c4e bytes) 000333 1780808440.796846 read 0x01aad9 thru 0x021b74 (0x709c bytes) 000335 1780808440.798777 write 0x00eee0 thru 0x0151f6 (0x6317 bytes) 000336 1780808440.803061 MIRROR_EXTEND to 2 mirrors 000337 1780808441.130247 mapwrite 0x00bbed thru 0x00d74b (0x1b5f bytes) 000338 1780808441.219537 MIRROR_RESYNC 2 mirrors 000339 1780808441.501828 mapwrite 0x039a76 thru 0x03ffff (0x658a bytes) 000341 1780808441.579648 mapread 0x00e2ea thru 0x01d38b (0xf0a2 bytes) 000342 1780808441.595582 mapread 0x03b3a3 thru 0x03ffff (0x4c5d bytes) 000343 1780808441.596735 trunc from 0x040000 to 0x0364e8 (0x9b17 bytes) 000345 1780808441.614206 trunc from 0x0364e9 to 0x025fce (0x1051a bytes) 000346 1780808441.628577 mapwrite 0x014697 thru 0x01d4f7 (0x8e61 bytes) 000347 1780808441.652892 mapwrite 0x037916 thru 0x03b9e2 (0x40cd bytes) 000349 1780808441.690588 mapwrite 0x029c2f thru 0x02d9fa (0x3dcc bytes) 000350 1780808441.710286 mapwrite 0x0262a7 thru 0x02ff4e (0x9ca8 bytes) 000351 1780808441.729769 MIRROR_RESYNC 2 mirrors 000354 1780808441.959511 MIRROR_RESYNC 2 mirrors 000355 1780808442.157655 read 0x0223c3 thru 0x02f7db (0xd419 bytes) 000358 1780808442.164910 mapread 0x0227fb thru 0x025e51 (0x3657 bytes) 000359 1780808442.165716 write 0x01da69 thru 0x021cfd (0x4295 bytes) 000363 1780808442.209071 read 0x027001 thru 0x0296fb (0x26fb bytes) 000365 1780808442.221526 write 0x001d3d thru 0x00d59d (0xb861 bytes) 000366 1780808442.225473 MIRROR_RESYNC 2 mirrors 000370 1780808442.490037 read 0x016d55 thru 0x01d473 (0x671f bytes) 000371 1780808442.496054 MIRROR_RESYNC 2 mirrors 000372 1780808442.722288 write 0x016399 thru 0x01888b (0x24f3 bytes) 000374 1780808442.781558 read 0x03404e thru 0x03b9e2 (0x7995 bytes) 000375 1780808442.802275 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000379 1780808442.882333 mapread 0x00e24e thru 0x013752 (0x5505 bytes) 000382 1780808442.906947 MIRROR_EXTEND to 2 mirrors 000383 1780808443.217223 mapwrite 0x01419f thru 0x01839e (0x4200 bytes) 000384 1780808443.331809 write 0x02c7fc thru 0x03667e (0x9e83 bytes) 000385 1780808443.344937 write 0x03763c thru 0x03ffff (0x89c4 bytes) 000386 1780808443.353399 mapwrite 0x021fce thru 0x023b25 (0x1b58 bytes) 000387 1780808443.400975 write 0x00d608 thru 0x016974 (0x936d bytes) 000389 1780808443.408087 trunc from 0x040000 to 0x01e403 (0x21bfc bytes) 000392 1780808443.426971 trunc from 0x01e404 to 0x005067 (0x1939c bytes) 000393 1780808443.435078 trunc from 0x005068 to 0x034be8 (0x2fb81 bytes) 000394 1780808443.442673 mapwrite 0x02edfd thru 0x03dc89 (0xee8d bytes) 000395 1780808443.517284 read 0x0081b4 thru 0x009dbb (0x1c08 bytes) 000396 1780808443.526374 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3dc8a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc8a): Mirror 2: 0x77289902 (stale) CRC-32 checksum value for chunk [0, 0x3dc8a): Mirror 1: 0x843c8f25 Mirror 3: 0x843c8f25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc8a: skipped 000397 1780808443.944803 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3dc8a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc8a): Mirror 2: 0x77289902 (stale) CRC-32 checksum value for chunk [0, 0x3dc8a): Mirror 1: 0x843c8f25 Mirror 3: 0x843c8f25 Mirror 4: 0x843c8f25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc8a: skipped 000399 1780808444.424127 mapwrite 0x0016e6 thru 0x00e250 (0xcb6b bytes) 000401 1780808444.537110 mapwrite 0x01c7b0 thru 0x02179e (0x4fef bytes) 000405 1780808444.572533 mapread 0x006dd8 thru 0x00cf49 (0x6172 bytes) 000406 1780808444.575952 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc8a): Mirror 3: 0x843c8f25 (stale) Mirror 4: 0x843c8f25 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc8a): Mirror 3: 0x843c8f25 (stale) Mirror 4: 0x843c8f25 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc8a: skipped 000408 1780808444.767072 read 0x028501 thru 0x0364bb (0xdfbb bytes) 000411 1780808444.772287 trunc from 0x03dc8a to 0x029053 (0x14c36 bytes) 000413 1780808444.802752 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29054): Mirror 4: 0x89c11b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29054): Mirror 4: 0x89c11b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29054: skipped 000414 1780808444.939462 write 0x02fd1a thru 0x036e25 (0x710c bytes) 000415 1780808444.941461 read 0x014108 thru 0x01b8a7 (0x77a0 bytes) 000416 1780808444.950640 mapread 0x007bce thru 0x0117e9 (0x9c1c bytes) 000418 1780808444.970873 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000421 1780808445.085771 MIRROR_EXTEND to 2 mirrors 000422 1780808445.414278 write 0x031d9c thru 0x0395ec (0x7851 bytes) 000423 1780808445.455484 mapwrite 0x00553c thru 0x010d11 (0xb7d6 bytes) 000425 1780808445.495889 write 0x011e4d thru 0x01bb85 (0x9d39 bytes) 000426 1780808445.506901 trunc from 0x0395ed to 0x02573e (0x13eae bytes) 000427 1780808445.521842 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000429 1780808445.635121 mapread 0x011c72 thru 0x019934 (0x7cc3 bytes) 000431 1780808445.649458 write 0x016d54 thru 0x01c18f (0x543c bytes) 000433 1780808445.653811 read 0x021847 thru 0x02573e (0x3ef8 bytes) 000436 1780808445.661517 trunc from 0x02573f to 0x007436 (0x1e308 bytes) 000438 1780808445.675633 read 0x000946 thru 0x007436 (0x6af1 bytes) 000441 1780808445.689488 mapwrite 0x03f2aa thru 0x03ffff (0xd56 bytes) 000442 1780808445.749700 write 0x014952 thru 0x01f9db (0xb08a bytes) 000445 1780808445.752664 read 0x03d00c thru 0x03ffff (0x2ff4 bytes) 000446 1780808445.753362 mapwrite 0x00ff70 thru 0x016164 (0x61f5 bytes) 000450 1780808445.795532 read 0x0215a3 thru 0x023434 (0x1e92 bytes) 000451 1780808445.795990 read 0x025345 thru 0x02c557 (0x7213 bytes) 000452 1780808445.796360 write 0x01c205 thru 0x0208ed (0x46e9 bytes) 000453 1780808445.797518 read 0x03e714 thru 0x03ffff (0x18ec bytes) 000454 1780808445.798192 read 0x00b3e6 thru 0x010795 (0x53b0 bytes) 000457 1780808445.800225 mapread 0x03d7b9 thru 0x03ffff (0x2847 bytes) 000458 1780808445.801714 read 0x011304 thru 0x011f4b (0xc48 bytes) 000459 1780808445.802382 write 0x0301d9 thru 0x03b243 (0xb06b bytes) 000460 1780808445.803345 read 0x0117e5 thru 0x01f340 (0xdb5c bytes) 000463 1780808445.804595 mapwrite 0x017ee3 thru 0x01cae1 (0x4bff bytes) 000464 1780808445.857195 MIRROR_EXTEND to 2 mirrors 000467 1780808446.262393 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000472 1780808446.378969 write 0x03217a thru 0x035c4f (0x3ad6 bytes) 000474 1780808446.398273 read 0x00e1a2 thru 0x013fb8 (0x5e17 bytes) 000475 1780808446.402966 write 0x01b23d thru 0x027ec3 (0xcc87 bytes) 000479 1780808446.423915 write 0x031eee thru 0x035aaa (0x3bbd bytes) 000483 1780808446.437123 mapwrite 0x03d080 thru 0x03ffff (0x2f80 bytes) 000484 1780808446.479253 mapwrite 0x01e50c thru 0x02ccd3 (0xe7c8 bytes) 000487 1780808446.504785 mapwrite 0x003ad0 thru 0x00d3a3 (0x98d4 bytes) 000488 1780808446.539987 trunc from 0x040000 to 0x023b35 (0x1c4ca bytes) 000492 1780808446.553963 mapwrite 0x0304f0 thru 0x03760b (0x711c bytes) 000495 1780808446.606932 trunc from 0x03760c to 0x026859 (0x10db2 bytes) 000496 1780808446.618374 read 0x006ee4 thru 0x010e82 (0x9f9f bytes) 000498 1780808446.620335 mapwrite 0x00a3c0 thru 0x01229e (0x7edf bytes) 000499 1780808446.659351 mapwrite 0x015013 thru 0x01ef39 (0x9f27 bytes) 000500 1780808446.686218 read 0x015b44 thru 0x017f77 (0x2434 bytes) 000502 1780808446.687331 write 0x014569 thru 0x020d6c (0xc804 bytes) 000504 1780808446.689253 mapwrite 0x00111a thru 0x00fef3 (0xedda bytes) 000505 1780808446.725565 write 0x03f351 thru 0x03ffff (0xcaf bytes) 000509 1780808446.728741 mapread 0x01a7dd thru 0x0211b7 (0x69db bytes) 000512 1780808446.734153 MIRROR_EXTEND to 2 mirrors 000513 1780808447.146499 mapread 0x02f83e thru 0x03e334 (0xeaf7 bytes) 000514 1780808447.164536 mapread 0x002dc6 thru 0x008850 (0x5a8b bytes) 000515 1780808447.171835 write 0x029692 thru 0x0344f7 (0xae66 bytes) 000516 1780808447.251085 mapread 0x02957f thru 0x02cf52 (0x39d4 bytes) 000517 1780808447.252261 read 0x007526 thru 0x013768 (0xc243 bytes) 000518 1780808447.260273 write 0x034e86 thru 0x03c260 (0x73db bytes) 000519 1780808447.264947 mapread 0x020cf1 thru 0x023c01 (0x2f11 bytes) 000520 1780808447.274462 mapwrite 0x009e97 thru 0x013800 (0x996a bytes) 000521 1780808447.310940 read 0x033510 thru 0x03ffff (0xcaf0 bytes) 000523 1780808447.324682 mapwrite 0x02556e thru 0x028dcb (0x385e bytes) 000524 1780808447.356136 trunc from 0x040000 to 0x02d88a (0x12775 bytes) 000528 1780808447.380065 mapwrite 0x00d37e thru 0x00df17 (0xb9a bytes) 000529 1780808447.400771 mapwrite 0x02e98f thru 0x02fc62 (0x12d4 bytes) 000530 1780808447.447665 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000531 1780808447.550848 MIRROR_EXTEND to 2 mirrors 000534 1780808447.823087 mapread 0x0015c7 thru 0x00acce (0x9708 bytes) 000535 1780808447.829696 mapread 0x015d3d thru 0x024bf9 (0xeebd bytes) 000538 1780808447.835595 write 0x006249 thru 0x01341e (0xd1d6 bytes) 000539 1780808447.862366 trunc from 0x02fc63 to 0x0376e6 (0x7a84 bytes) 000540 1780808447.878045 read 0x027283 thru 0x02d1fa (0x5f78 bytes) 000542 1780808447.884892 trunc from 0x0376e7 to 0x024f55 (0x12791 bytes) 000543 1780808447.901143 read 0x01e6fe thru 0x01faab (0x13ae bytes) 000544 1780808447.908025 read 0x01563d thru 0x01b62a (0x5fee bytes) 000545 1780808447.911188 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24f56): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24f56): Mirror 2: 0x18c2db8b (stale) CRC-32 checksum value for chunk [0, 0x24f56): Mirror 1: 0x34aca9ff Mirror 3: 0x34aca9ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24f56: skipped 000546 1780808448.304282 mapread 0x006559 thru 0x0144c1 (0xdf69 bytes) truncating to largest ever: 0x3fb48 000547 1780808448.329208 trunc from 0x024f56 to 0x03fb47 (0x1abf2 bytes) 000550 1780808448.363280 mapread 0x011fb9 thru 0x01b9fe (0x9a46 bytes) 000551 1780808448.376917 MIRROR_RESYNC 3 mirrors 000554 1780808448.782043 write 0x0297e3 thru 0x038e33 (0xf651 bytes) 000555 1780808448.834700 MIRROR_RESYNC 3 mirrors 000556 1780808449.238295 trunc from 0x03fb48 to 0x00f16b (0x309dc bytes) 000558 1780808449.285816 trunc from 0x00f16c to 0x0307cc (0x21661 bytes) 000560 1780808449.302575 write 0x001eb0 thru 0x004e36 (0x2f87 bytes) 000562 1780808449.312257 write 0x0212cc thru 0x023b73 (0x28a8 bytes) 000567 1780808449.322510 write 0x007d51 thru 0x00d5a1 (0x5851 bytes) 000569 1780808449.339711 MIRROR_RESYNC 3 mirrors 000570 1780808449.635449 write 0x004c57 thru 0x011f7a (0xd324 bytes) 000571 1780808449.660323 mapwrite 0x01833f thru 0x020ba2 (0x8864 bytes) 000572 1780808449.700373 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x307cd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x307cd): Mirror 2: 0xb858d410 (stale) Mirror 3: 0xb858d410 (stale) CRC-32 checksum value for chunk [0, 0x307cd): Mirror 1: 0xd2d17df4 Mirror 4: 0xd2d17df4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x307cd: skipped 000573 1780808450.094354 mapwrite 0x0065d2 thru 0x015d08 (0xf737 bytes) 000575 1780808450.180985 read 0x02345f thru 0x0255e9 (0x218b bytes) 000576 1780808450.186380 write 0x003ca4 thru 0x01012f (0xc48c bytes) 000577 1780808450.191233 MIRROR_RESYNC 4 mirrors 000579 1780808450.560668 write 0x031452 thru 0x03a55e (0x910d bytes) 000580 1780808450.587075 mapread 0x016a0f thru 0x01a8e3 (0x3ed5 bytes) 000585 1780808450.600769 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a55f): Mirror 2: 0x8abd120e (stale) Mirror 3: 0x8abd120e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a55f): Mirror 2: 0x8abd120e (stale) Mirror 3: 0x8abd120e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a55f: skipped 000587 1780808450.781277 mapwrite 0x013c38 thru 0x01d053 (0x941c bytes) 000588 1780808450.804074 write 0x01f79c thru 0x02ce80 (0xd6e5 bytes) 000589 1780808450.811023 mapread 0x02c45c thru 0x0374cf (0xb074 bytes) 000591 1780808450.817963 trunc from 0x03a55f to 0x020a6d (0x19af1 bytes) 000593 1780808450.831853 write 0x01d30a thru 0x02b63c (0xe333 bytes) 000595 1780808450.834049 mapread 0x028b01 thru 0x02b63c (0x2b3c bytes) 000599 1780808450.837008 mapwrite 0x02ede9 thru 0x037b50 (0x8d68 bytes) 000600 1780808450.887551 write 0x02c45e thru 0x03b777 (0xf31a bytes) 000601 1780808450.891757 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b778): Mirror 3: 0xeee69583 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b778): Mirror 3: 0xeee69583 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b778: skipped 000603 1780808451.038301 trunc from 0x03b778 to 0x019b21 (0x21c56 bytes) 000605 1780808451.050975 trunc from 0x019b22 to 0x003893 (0x1628e bytes) 000606 1780808451.068330 write 0x019e03 thru 0x022c4b (0x8e49 bytes) 000607 1780808451.073828 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000611 1780808451.145824 write 0x01654d thru 0x01ce65 (0x6919 bytes) 000612 1780808451.151163 mapwrite 0x007d2f thru 0x00bd9b (0x406d bytes) 000614 1780808451.180988 mapwrite 0x023fc3 thru 0x026eba (0x2ef8 bytes) 000615 1780808451.208342 MIRROR_EXTEND to 2 mirrors 000616 1780808451.459011 trunc from 0x026ebb to 0x01290a (0x145b0 bytes) 000617 1780808451.535154 mapwrite 0x00c5b2 thru 0x012ff1 (0x6a40 bytes) 000618 1780808451.609924 mapread 0x00c31a thru 0x012ff1 (0x6cd8 bytes) 000619 1780808451.611253 read 0x00547f thru 0x007403 (0x1f85 bytes) 000620 1780808451.615459 mapread 0x011dcd thru 0x012ff1 (0x1225 bytes) 000621 1780808451.616319 read 0x009393 thru 0x012ff1 (0x9c5f bytes) 000623 1780808451.617288 mapwrite 0x03b4a9 thru 0x03ffff (0x4b57 bytes) 000626 1780808451.677922 trunc from 0x040000 to 0x022587 (0x1da78 bytes) 000629 1780808451.692509 mapwrite 0x035900 thru 0x03cc57 (0x7358 bytes) 000630 1780808451.786677 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000633 1780808451.873621 write 0x01f595 thru 0x02219c (0x2c08 bytes) 000634 1780808451.879745 mapread 0x02a21d thru 0x0377c8 (0xd5ac bytes) 000635 1780808451.892824 trunc from 0x03cc58 to 0x035e6a (0x6ded bytes) 000637 1780808451.903344 write 0x00011c thru 0x001a8f (0x1974 bytes) 000639 1780808451.909330 write 0x03330e thru 0x0359af (0x26a2 bytes) 000640 1780808451.913235 MIRROR_EXTEND to 2 mirrors 000643 1780808452.160863 mapread 0x00f671 thru 0x01aa5c (0xb3ec bytes) 000645 1780808452.179248 MIRROR_RESYNC 2 mirrors 000647 1780808452.358361 write 0x032002 thru 0x03ffff (0xdffe bytes) 000648 1780808452.390463 mapwrite 0x012489 thru 0x014e4d (0x29c5 bytes) 000649 1780808452.429999 mapwrite 0x010518 thru 0x01d84c (0xd335 bytes) 000650 1780808452.472747 read 0x0320ec thru 0x03ffff (0xdf14 bytes) 000653 1780808452.474724 mapread 0x02577d thru 0x02c013 (0x6897 bytes) 000654 1780808452.480588 mapwrite 0x02ccb7 thru 0x0362aa (0x95f4 bytes) 000659 1780808452.526723 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000660 1780808452.606652 read 0x003a5d thru 0x00fedc (0xc480 bytes) 000662 1780808452.622715 mapwrite 0x002ac8 thru 0x010d75 (0xe2ae bytes) 000665 1780808452.680465 mapwrite 0x039654 thru 0x03ffff (0x69ac bytes) 000667 1780808452.713247 mapread 0x032cee thru 0x037a87 (0x4d9a bytes) 000668 1780808452.714330 write 0x01f9d8 thru 0x028ecf (0x94f8 bytes) 000669 1780808452.722012 write 0x032ad6 thru 0x03bd39 (0x9264 bytes) 000670 1780808452.722813 read 0x01f2e7 thru 0x029e02 (0xab1c bytes) 000672 1780808452.727954 read 0x026f03 thru 0x033cd4 (0xcdd2 bytes) 000673 1780808452.731302 MIRROR_EXTEND to 2 mirrors 000674 1780808453.037915 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000675 1780808453.107726 mapwrite 0x03d46c thru 0x03ffff (0x2b94 bytes) 000678 1780808453.153371 mapread 0x030423 thru 0x030592 (0x170 bytes) 000680 1780808453.160720 mapwrite 0x00af55 thru 0x00d986 (0x2a32 bytes) 000681 1780808453.185191 write 0x00e458 thru 0x01792c (0x94d5 bytes) 000686 1780808453.192079 write 0x039e45 thru 0x03ffff (0x61bb bytes) 000688 1780808453.193964 mapwrite 0x038770 thru 0x03ffff (0x7890 bytes) 000693 1780808453.222502 read 0x0352e7 thru 0x038cc0 (0x39da bytes) 000694 1780808453.223101 mapread 0x014f32 thru 0x019ef5 (0x4fc4 bytes) 000695 1780808453.227981 mapwrite 0x00c8a7 thru 0x017472 (0xabcc bytes) 000696 1780808453.249708 mapread 0x039d6b thru 0x03ffff (0x6295 bytes) 000697 1780808453.250709 trunc from 0x040000 to 0x029f56 (0x160a9 bytes) 000698 1780808453.278316 mapwrite 0x00cd2b thru 0x01635d (0x9633 bytes) 000699 1780808453.302148 write 0x02f07a thru 0x03af15 (0xbe9c bytes) 000702 1780808453.305354 read 0x0317eb thru 0x03af15 (0x972b bytes) 000706 1780808453.307979 mapread 0x034ff4 thru 0x03af15 (0x5f22 bytes) 000707 1780808453.309152 read 0x023709 thru 0x0320ba (0xe9b2 bytes) 000708 1780808453.317365 write 0x014f8a thru 0x023b32 (0xeba9 bytes) 000709 1780808453.318690 mapwrite 0x00a3bd thru 0x0173ef (0xd033 bytes) 000711 1780808453.354222 write 0x03e527 thru 0x03ffff (0x1ad9 bytes) 000716 1780808453.357272 read 0x03291c thru 0x03add8 (0x84bd bytes) 000717 1780808453.358014 write 0x02611c thru 0x0335ec (0xd4d1 bytes) 000719 1780808453.360307 trunc from 0x040000 to 0x022ec7 (0x1d138 bytes) 000722 1780808453.379240 MIRROR_EXTEND to 2 mirrors 000724 1780808453.680124 write 0x0269b9 thru 0x031252 (0xa89a bytes) 000725 1780808453.723497 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31253): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31253): Mirror 2: 0x65388d8b (stale) CRC-32 checksum value for chunk [0, 0x31253): Mirror 1: 0xa37932e8 Mirror 3: 0xa37932e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31253: skipped 000726 1780808454.073984 read 0x020f95 thru 0x02ad03 (0x9d6f bytes) 000727 1780808454.079352 mapread 0x02deca thru 0x031252 (0x3389 bytes) 000731 1780808454.101639 MIRROR_RESYNC 3 mirrors 000732 1780808454.460893 MIRROR_EXTEND to 4 mirrors 000733 1780808454.879537 mapread 0x011bd4 thru 0x011c26 (0x53 bytes) 000737 1780808454.885186 read 0x00516d thru 0x00624d (0x10e1 bytes) 000738 1780808454.891928 read 0x027d8e thru 0x031252 (0x94c5 bytes) 000743 1780808454.900713 mapwrite 0x00a3bb thru 0x011e87 (0x7acd bytes) 000745 1780808455.019391 mapwrite 0x02934f thru 0x030d35 (0x79e7 bytes) 000748 1780808455.087662 write 0x03f426 thru 0x03ffff (0xbda bytes) 000749 1780808455.089048 trunc from 0x040000 to 0x0267a8 (0x19857 bytes) 000750 1780808455.112040 read 0x018243 thru 0x01ad7a (0x2b38 bytes) 000752 1780808455.127274 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x267a9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x267a9): Mirror 2: 0x26f579bc (stale) Mirror 3: 0x26f579bc (stale) Mirror 4: 0x26f579bc (stale) CRC-32 checksum value for chunk [0, 0x267a9): Mirror 1: 0xd0f894fe Mirror 5: 0xd0f894fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x267a9: skipped 000754 1780808455.529850 read 0x003a9a thru 0x00babc (0x8023 bytes) 000755 1780808455.537308 mapread 0x000ad4 thru 0x00252a (0x1a57 bytes) 000757 1780808455.550362 trunc from 0x0267a9 to 0x01c195 (0xa613 bytes) 000758 1780808455.603825 read 0x00cccb thru 0x013c6e (0x6fa4 bytes) 000763 1780808455.616858 read 0x01876d thru 0x01c12e (0x39c2 bytes) 000765 1780808455.618203 trunc from 0x01c196 to 0x02f881 (0x136ec bytes) 000766 1780808455.640875 write 0x02bf19 thru 0x02f090 (0x3178 bytes) 000771 1780808455.669761 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f882): Mirror 3: 0xbe3ef63c (stale) Mirror 4: 0xbe3ef63c (stale) Mirror 5: 0x119e10d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f882): Mirror 3: 0xbe3ef63c (stale) Mirror 4: 0xbe3ef63c (stale) Mirror 5: 0x119e10d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f882: skipped 000772 1780808455.837441 read 0x000b41 thru 0x009afc (0x8fbc bytes) 000774 1780808455.850689 MIRROR_RESYNC 4 mirrors 000775 1780808456.343695 mapread 0x01b742 thru 0x026608 (0xaec7 bytes) 000776 1780808456.364228 mapread 0x02da7e thru 0x02f881 (0x1e04 bytes) 000777 1780808456.367829 read 0x017fc9 thru 0x01e233 (0x626b bytes) 000778 1780808456.368107 read 0x003391 thru 0x008461 (0x50d1 bytes) 000779 1780808456.373249 trunc from 0x02f882 to 0x018947 (0x16f3a bytes) 000781 1780808456.409776 write 0x03f7ee thru 0x03ffff (0x812 bytes) 000782 1780808456.410646 write 0x001bcf thru 0x0094cb (0x78fd bytes) 000783 1780808456.417407 trunc from 0x040000 to 0x01dbe9 (0x22416 bytes) 000784 1780808456.429277 mapread 0x00f14b thru 0x0131c6 (0x407c bytes) 000785 1780808456.442189 write 0x03d9fa thru 0x03ffff (0x2606 bytes) 000787 1780808456.447256 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x264bcb00 (stale) Mirror 4: 0x264bcb00 (stale) Mirror 5: 0x264bcb00 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbdca767 Mirror 6: 0xbdca767 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000788 1780808456.927325 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x264bcb00 (stale) Mirror 4: 0x264bcb00 (stale) Mirror 5: 0x264bcb00 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbdca767 Mirror 6: 0xbdca767 Mirror 7: 0xbdca767 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000789 1780808457.382316 write 0x007c21 thru 0x0138cd (0xbcad bytes) 000793 1780808457.450890 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x264bcb00 (stale) Mirror 4: 0x264bcb00 (stale) Mirror 6: 0xbdca767 (stale) Mirror 7: 0xbdca767 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x264bcb00 (stale) Mirror 4: 0x264bcb00 (stale) Mirror 6: 0xbdca767 (stale) Mirror 7: 0xbdca767 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000796 1780808457.725743 MIRROR_RESYNC 5 mirrors 000804 1780808458.266700 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000805 1780808458.488189 trunc from 0x040000 to 0x019961 (0x2669e bytes) 000806 1780808458.528616 mapwrite 0x03a048 thru 0x03ffff (0x5fb8 bytes) 000810 1780808458.596429 mapread 0x024317 thru 0x02b025 (0x6d0f bytes) 000811 1780808458.602640 mapread 0x03f5d8 thru 0x03ffff (0xa28 bytes) 000812 1780808458.606686 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x99f66755 (stale) Mirror 6: 0x99f66755 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x99f66755 (stale) Mirror 6: 0x99f66755 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000815 1780808458.800209 mapread 0x0361c0 thru 0x039dd9 (0x3c1a bytes) 000816 1780808458.812292 trunc from 0x040000 to 0x01f1cb (0x20e34 bytes) 000817 1780808458.821973 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f1cc): Mirror 3: 0x528670c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f1cc): Mirror 3: 0x528670c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f1cc: skipped 000818 1780808458.991586 mapwrite 0x03c01d thru 0x03ffff (0x3fe3 bytes) 000819 1780808459.060857 read 0x00839f thru 0x008ab7 (0x719 bytes) 000820 1780808459.070002 mapwrite 0x038dec thru 0x03ffff (0x7214 bytes) 000821 1780808459.093216 trunc from 0x040000 to 0x0376d6 (0x8929 bytes) 000825 1780808459.102797 read 0x0181aa thru 0x020490 (0x82e7 bytes) 000827 1780808459.112606 mapwrite 0x0389f4 thru 0x03ffff (0x760c bytes) 000828 1780808459.159470 mapread 0x0137bd thru 0x022003 (0xe847 bytes) 000829 1780808459.160504 mapwrite 0x023813 thru 0x02c2de (0x8acc bytes) 000830 1780808459.182347 mapwrite 0x000539 thru 0x00a0a5 (0x9b6d bytes) 000832 1780808459.205154 mapwrite 0x03a9c0 thru 0x03ffff (0x5640 bytes) 000833 1780808459.227257 read 0x000755 thru 0x00df3a (0xd7e6 bytes) 000834 1780808459.227952 mapwrite 0x038eaf thru 0x039fad (0x10ff bytes) 000836 1780808459.248846 mapwrite 0x0196ec thru 0x01a761 (0x1076 bytes) 000837 1780808459.274149 trunc from 0x040000 to 0x01727a (0x28d85 bytes) 000838 1780808459.285211 trunc from 0x01727b to 0x008f96 (0xe2e4 bytes) 000839 1780808459.301673 write 0x03f620 thru 0x03ffff (0x9e0 bytes) 000840 1780808459.307157 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000841 1780808459.399943 mapread 0x012e56 thru 0x0211a8 (0xe353 bytes) 000842 1780808459.419481 mapwrite 0x01cc68 thru 0x0287ae (0xbb47 bytes) 000843 1780808459.464289 read 0x037ef2 thru 0x03da64 (0x5b73 bytes) 000845 1780808459.469802 mapwrite 0x005dce thru 0x00a143 (0x4376 bytes) 000850 1780808459.521351 trunc from 0x040000 to 0x030fc9 (0xf036 bytes) 000853 1780808459.546594 MIRROR_EXTEND to 2 mirrors 000854 1780808459.860941 mapwrite 0x03d503 thru 0x03ffff (0x2afd bytes) 000856 1780808459.955390 write 0x03a5c5 thru 0x03ffff (0x5a3b bytes) 000857 1780808459.956278 write 0x02eec5 thru 0x03a85c (0xb998 bytes) 000859 1780808459.962290 write 0x015cc8 thru 0x023740 (0xda79 bytes) 000860 1780808459.969982 mapwrite 0x0286ad thru 0x02a03e (0x1992 bytes) 000861 1780808460.006940 MIRROR_RESYNC 2 mirrors 000862 1780808460.409630 read 0x0270df thru 0x033313 (0xc235 bytes) 000863 1780808460.414310 read 0x011dc3 thru 0x020211 (0xe44f bytes) 000865 1780808460.422455 trunc from 0x040000 to 0x00bdfd (0x34202 bytes) 000866 1780808460.475824 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000867 1780808460.540023 write 0x022737 thru 0x026ea8 (0x4772 bytes) 000869 1780808460.542543 mapwrite 0x0164f5 thru 0x0198e7 (0x33f3 bytes) 000870 1780808460.578080 read 0x01d528 thru 0x01f6bc (0x2195 bytes) 000872 1780808460.580047 trunc from 0x026ea9 to 0x03af5a (0x140b2 bytes) 000874 1780808460.588002 mapread 0x0047df thru 0x009f56 (0x5778 bytes) 000875 1780808460.595498 MIRROR_EXTEND to 2 mirrors 000877 1780808460.923401 MIRROR_EXTEND to 3 mirrors 000880 1780808461.358610 MIRROR_RESYNC 3 mirrors 000885 1780808461.718347 trunc from 0x03af5b to 0x00fa97 (0x2b4c3 bytes) 000886 1780808461.776528 mapread 0x00e28f thru 0x00fa97 (0x1809 bytes) 000889 1780808461.786810 mapread 0x00cd97 thru 0x00fa97 (0x2d01 bytes) 000891 1780808461.795959 trunc from 0x00fa98 to 0x03dc66 (0x2e1cf bytes) 000894 1780808461.815908 write 0x02d0a2 thru 0x03010c (0x306b bytes) 000895 1780808461.824990 trunc from 0x03dc67 to 0x009a1c (0x3424a bytes) 000897 1780808461.838561 trunc from 0x009a1d to 0x03ed90 (0x35374 bytes) 000898 1780808461.867701 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed91): Mirror 2: 0x86f02c9a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed91): Mirror 2: 0x86f02c9a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed91: skipped 000901 1780808462.088893 mapwrite 0x003700 thru 0x0074fa (0x3dfb bytes) 000902 1780808462.139792 write 0x00ad4e thru 0x011e1f (0x70d2 bytes) 000903 1780808462.146586 read 0x01d9be thru 0x02d250 (0xf893 bytes) 000904 1780808462.161486 mapwrite 0x003156 thru 0x003b8f (0xa3a bytes) 000907 1780808462.204100 read 0x03b23e thru 0x03ed90 (0x3b53 bytes) 000908 1780808462.231769 MIRROR_RESYNC 2 mirrors 000909 1780808462.662108 trunc from 0x03ed91 to 0x037942 (0x744e bytes) 000911 1780808462.736085 trunc from 0x037943 to 0x01a38e (0x1d5b4 bytes) 000912 1780808462.753319 mapread 0x012928 thru 0x015060 (0x2739 bytes) 000915 1780808462.773935 trunc from 0x01a38f to 0x03691e (0x1c590 bytes) 000916 1780808462.790871 write 0x031deb thru 0x03ffff (0xe215 bytes) 000917 1780808462.795412 trunc from 0x040000 to 0x02e4ba (0x11b45 bytes) 000918 1780808462.818700 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) 000919 1780808462.938581 mapwrite 0x014566 thru 0x019780 (0x521b bytes) 000920 1780808463.007313 read 0x012bf9 thru 0x0218c8 (0xecd0 bytes) 000921 1780808463.014654 mapwrite 0x039e75 thru 0x03ffff (0x618b bytes) 000924 1780808463.082137 read 0x03dd9f thru 0x03ffff (0x2261 bytes) 000926 1780808463.084122 write 0x031dcb thru 0x03b30c (0x9542 bytes) 000929 1780808463.097883 write 0x02d1de thru 0x0343ff (0x7222 bytes) 000931 1780808463.100737 mapread 0x01b160 thru 0x01b1a5 (0x46 bytes) 000932 1780808463.108691 MIRROR_EXTEND to 2 mirrors 000934 1780808463.495055 read 0x00d104 thru 0x0104b9 (0x33b6 bytes) 000941 1780808463.500348 mapread 0x001369 thru 0x008b2b (0x77c3 bytes) 000942 1780808463.510366 mapread 0x00deb9 thru 0x01acf2 (0xce3a bytes) 000943 1780808463.523101 MIRROR_EXTEND to 3 mirrors 000944 1780808463.901647 mapread 0x01e11b thru 0x02714c (0x9032 bytes) 000946 1780808463.944180 trunc from 0x040000 to 0x03b343 (0x4cbc bytes) 000950 1780808464.035449 mapread 0x014a83 thru 0x0167b9 (0x1d37 bytes) 000951 1780808464.042101 read 0x008274 thru 0x00f9af (0x773c bytes) 000953 1780808464.049910 trunc from 0x03b344 to 0x008faf (0x32394 bytes) 000954 1780808464.067059 trunc from 0x008fb0 to 0x02f423 (0x26474 bytes) 000956 1780808464.086148 mapread 0x00161a thru 0x005b16 (0x44fd bytes) 000957 1780808464.094371 mapwrite 0x038aaf thru 0x03e62a (0x5b7c bytes) 000958 1780808464.159861 mapwrite 0x00f58b thru 0x0102f7 (0xd6d bytes) 000959 1780808464.191366 write 0x038674 thru 0x03ffff (0x798c bytes) 000960 1780808464.192568 trunc from 0x040000 to 0x00fd4d (0x302b2 bytes) 000962 1780808464.210405 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x3b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfd4e): Mirror 2: 0xe33a5b1a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfd4e): Mirror 2: 0xe33a5b1a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfd4e: skipped 000963 1780808464.395056 mapwrite 0x01dd60 thru 0x0286ec (0xa98d bytes) 000965 1780808464.465330 write 0x02d010 thru 0x035900 (0x88f1 bytes) 000968 1780808464.467321 mapwrite 0x03570f thru 0x03ffff (0xa8f1 bytes) 000969 1780808464.517263 read 0x01a682 thru 0x0298a3 (0xf222 bytes) 000970 1780808464.519969 trunc from 0x040000 to 0x02c274 (0x13d8b bytes) 000971 1780808464.528887 mapwrite 0x00d2ad thru 0x01213c (0x4e90 bytes) 000973 1780808464.560543 write 0x01a875 thru 0x01c2e2 (0x1a6e bytes) 000974 1780808464.561360 mapwrite 0x007137 thru 0x00ee36 (0x7d00 bytes) 000975 1780808464.598056 mapwrite 0x0254f6 thru 0x02afda (0x5ae5 bytes) 000976 1780808464.638604 trunc from 0x02c275 to 0x015846 (0x16a2e bytes) 000979 1780808464.658251 read 0x00bac4 thru 0x015846 (0x9d83 bytes) 000980 1780808464.660343 read 0x00f930 thru 0x015846 (0x5f17 bytes) 000982 1780808464.665690 mapread 0x011087 thru 0x015846 (0x47c0 bytes) 000984 1780808464.667071 trunc from 0x015847 to 0x02f523 (0x19cdd bytes) 000989 1780808464.687375 read 0x027017 thru 0x02f523 (0x850d bytes) 000995 1780808464.704134 trunc from 0x02f524 to 0x012ccc (0x1c857 bytes) 000996 1780808464.722019 mapwrite 0x0038be thru 0x00c02f (0x8772 bytes) 000999 1780808464.766659 write 0x00160f thru 0x001ca9 (0x69b bytes) 001000 1780808464.774845 MIRROR_RESYNC 2 mirrors All operations completed A-OK! PASS 70a (38s) == sanity-flr test 71: check mirror extend parent fid ==== 01:01:07 (1780808467) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 01:01:08 (1780808468) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x591:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105847 s, 19.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x591:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x592:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057751 s, 18.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x591:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x592:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5b1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x591:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x592:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5b1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x593:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 01:01:12 (1780808472) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5530656 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4287840 .. Extending file size to 6857408 .. Extending file size to 8385792 .. Extending file size to 8344448 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8401728 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8996320 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8491776 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8722528 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9141344 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9005856 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 60115 60116 60118 60119 60120 done 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) 192.168.202.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:02:19 (1780808539) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6462944 .. Extending file size to 6850912 .. Extending file size to 8732608 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8942432 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9184064 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9299328 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8900192 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 62929 62930 62932 62934 62936 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) 192.168.202.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:03:24 (1780808604) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3456 2204928 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 26624 3742720 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 45056 7493632 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3456 2204928 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 2432 2206208 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 26624 3742720 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 45056 7493632 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0362656 s, 57.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0764703 s, 27.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe205fb85 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:03:30 (1780808610) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0294721 s, 71.2 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 01:03:32 (1780808612) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x39e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5c7:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5c7:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5c8:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5a8:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:03:35 (1780808615) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0525666 s, 39.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x56:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x58:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x59:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:03:37 (1780808617) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0503713 s, 41.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5ca:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5a9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5ca:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5a9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5aa:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:03:40 (1780808620) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0377698 s, 55.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5b:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:04:04 (1780808644) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316962 s, 33.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x58:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x58:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x59:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:04:49 (1780808689) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0421159 s, 49.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5cc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5ab:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5cc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5ab:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:05:33 (1780808733) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg209-server: Pool lustre.test_204f created Adding targets to pool oleg209-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg209-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0496044 s, 42.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x5cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg209-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg209-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg209-server: Pool lustre.test_204f destroyed PASS 204f (56s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:06:29 (1780808789) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:06:31 (1780808791) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:06:33 (1780808793) oleg209-server: Pool lustre.test_206 created Adding targets to pool oleg209-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg209-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg209-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg209-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg209-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:06:46 (1780808806) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269241 s, 38.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116907 s, 89.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf9c3360b Mirror 3: 0xf9c3360b PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:06:49 (1780808809) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:06:50 (1780808810) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:06:51 (1780808811) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0277407 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xac0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x670000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x670000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x670000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1730000 Read this (4 bytes): BBBB 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) PASS 209a (17s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:07:08 (1780808828) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00604822 s, 1.4 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:07:10 (1780808830) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161915 s, 64.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x3bd lmm_fid: [0x200000405:0x3bd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x3bd lmm_fid: [0x200000405:0x3bd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x5b7:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339612093 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-07 01:07:10.000000000 -0400 Modify: 2026-06-07 01:07:10.000000000 -0400 Change: 2026-06-07 01:07:11.000000000 -0400 Birth: 2026-06-07 01:07:11.000000000 -0400 PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:07:12 (1780808832) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:07:13 (1780808833) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.11959 s, 343 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0123729 s, 331 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 7 01:07 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3bf:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 7 01:07 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:07:16 (1780808836) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:07:18 (1780808838) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (2s) == sanity-flr test complete, duration 939 sec ============ 01:07:21 (1780808841) === sanity-flr: start cleanup 01:07:21 (1780808841) === === sanity-flr: finish cleanup 01:07:23 (1780808843) ===