-----============= acceptance-small: sanity-flr ============----- Mon Sep 7 21:22:13 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg148-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 21:22:33 (1788830553) === oleg148-client.virtnet: executing check_config_client /mnt/lustre oleg148-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg148-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ce9c2ddb000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ce9c2ddb000.idle_timeout=debug disable quota as required oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 21:23:06 (1788830586) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7653] == sanity-flr test 0a: lfs mirror create with -N option == 21:23:08 (1788830588) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:23:22 (1788830602) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:23:25 (1788830605) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:23:28 (1788830608) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:23:50 (1788830630) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:23:52 (1788830632) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 21:23:55 (1788830635) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8ce9c2ddb000.stats=clear osc.lustre-OST0001-osc-ffff8ce9c2ddb000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:24:02 (1788830642) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:24:11 (1788830651) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:24:18 (1788830658) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:24:21 (1788830661) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:24:28 (1788830668) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 21:24:35 (1788830675) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:24:42 (1788830682) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:24:44 (1788830684) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:24:50 (1788830690) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:24:59 (1788830699) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:25:05 (1788830705) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.76096 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978988 s, 10.7 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:25:14 (1788830714) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.264941 s, 55.4 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:25:21 (1788830721) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:25:29 (1788830729) Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ce9c2ddb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ce9c2ddb000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ce9c2ddb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ce9c2ddb000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (60s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:26:29 (1788830789) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136246 s, 15.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ce9c2ddb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ce9c2ddb000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (32s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:27:01 (1788830821) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ce9c2ddb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ce9c2ddb000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ce9c2ddb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ce9c2ddb000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (103s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:28:44 (1788830924) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.45542 s, 8.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 7 21:28 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.258s user 0m0.001s sys 0m0.213s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ce9c2ddb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ce9c2ddb000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.035s user 0m0.001s sys 0m0.172s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ce9c2ddb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ce9c2ddb000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.126s user 0m0.001s sys 0m0.179s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ce9c2ddb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ce9c2ddb000.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:30:18 (1788831018) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:30:22 (1788831022) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:30:25 (1788831025) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 21:30:28 (1788831028) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 21:30:36 (1788831036) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.595246 s, 21.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:30:50 (1788831050) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439002 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448036 s, 23.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:30:58 (1788831058) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:31:05 (1788831065) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0007668 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0088768 s, 115 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000845115 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0040209 s, 255 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 21:31:17 (1788831077) create test files with size 15752995 .. d328d3bcccd585a40cb11b08bf565aba - 19f161b4e3ceba54bb7a8892769f5fb3 - 8f24ef47d3c8ad84cc8607806908c0a5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 21:31:44 (1788831104) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:32:00 (1788831120) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:32:06 (1788831126) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931798 s, 11.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02556 s, 41.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 21:32:15 (1788831135) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145079 s, 14.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.326326 s, 12.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.313314 s, 6.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 21:32:31 (1788831151) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 21:32:34 (1788831154) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:32:38 (1788831158) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516422 s, 20.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 21:32:46 (1788831166) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:32:49 (1788831169) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:33:02 (1788831182) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.44888 s, 23.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:33:16 (1788831196) llite.lustre-ffff8ce9ce492000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.512498 s, 20.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8ce9c8a32000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:33:34 (1788831214) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:34:12 (1788831252) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:34:48 (1788831288) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 21:34:51 (1788831291) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.42636 s, 7.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.128018 s, 24.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x75b6f0f7 (nosync,stale) Mirror 2: 0x75b6f0f7 (nosync,stale) ** verify mirror 2 doesn't change original checksum: f010319b786d426306b615aae9565569 - mirror 1 checksum: 56b67efb7ccac8bdc4aa037958e3c996 - mirror 2 checksum: f010319b786d426306b615aae9565569 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T21:34:54-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T21:34:54-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f010319b786d426306b615aae9565569 - mirror 1 checksum: 56b67efb7ccac8bdc4aa037958e3c996 - mirror 2 checksum: 56b67efb7ccac8bdc4aa037958e3c996 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T21:34:54-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T21:34:54-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:35:03 (1788831303) Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 21:35:11 (1788831311) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 21:35:35 (1788831335) targets are mounted 21:35:35 (1788831335) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219527 s, 47.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:35:52 (1788831352) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:35:56 (1788831356) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:35:59 (1788831359) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0138975 s, 1.2 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:36:06 (1788831366) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:36:10 (1788831370) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0127303 s, 322 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0689923 s, 59.4 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:36:18 (1788831378) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:36:45 (1788831405) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:37:15 (1788831435) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:37:41 (1788831461) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:38:06 (1788831486) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 21:38:14 (1788831494) mirror create pid 54346 mirror split pid 54347 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54346 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54347 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 21:39:20 (1788831560) Seed set to 1562 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1788831562.544270 write 0x036790 thru 0x03803c (0x18ad bytes) 000004 1788831562.552941 write 0x010c89 thru 0x0207e9 (0xfb61 bytes) 000005 1788831562.578448 read 0x021121 thru 0x022b63 (0x1a43 bytes) truncating to largest ever: 0x2c43d 000008 1788831562.593122 trunc from 0x03803d to 0x02c43c (0xbc00 bytes) 000009 1788831562.624598 read 0x01babe thru 0x0265d4 (0xab17 bytes) truncating to largest ever: 0x3fcbf 000010 1788831562.638276 trunc from 0x02c43d to 0x03fcbe (0x13882 bytes) 000011 1788831562.660149 mapwrite 0x03500f thru 0x03ffff (0xaff1 bytes) 000012 1788831562.745631 read 0x0355e1 thru 0x03931c (0x3d3c bytes) 000013 1788831562.746264 write 0x03d2bc thru 0x03ffff (0x2d44 bytes) 000017 1788831562.749188 write 0x02d883 thru 0x03299a (0x5118 bytes) 000018 1788831562.750501 punch from 0x024c48 to 0x02e4f3 (0x98ac bytes) 000021 1788831562.767977 mapwrite 0x01b8bd thru 0x024ca0 (0x93e4 bytes) 000022 1788831562.843228 trunc from 0x040000 to 0x01cd4c (0x232b3 bytes) 000024 1788831562.868689 punch from 0x01134a to 0x01cb6c (0xb823 bytes) 000025 1788831562.878128 trunc from 0x01cd4d to 0x01175a (0xb5f2 bytes) 000026 1788831562.898964 punch from 0x0033a2 to 0x006e53 (0x3ab2 bytes) 000027 1788831562.909176 write 0x03a631 thru 0x03ffff (0x59cf bytes) 000029 1788831562.910733 mapwrite 0x00a3da thru 0x00b91d (0x1544 bytes) 000032 1788831562.951135 trunc from 0x040000 to 0x01ea0d (0x215f2 bytes) 000033 1788831562.996064 MIRROR_EXTEND to 2 mirrors 000034 1788831563.825223 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000035 1788831564.000818 punch from 0x017446 to 0x01be4b (0x4a06 bytes) 000037 1788831564.027190 trunc from 0x01ea0e to 0x00fa23 (0xefea bytes) 000039 1788831564.043273 punch from 0x00e4e5 to 0x00fa23 (0x153f bytes) 000041 1788831564.047823 mapread 0x004cb8 thru 0x00fa23 (0xad6c bytes) 000043 1788831564.069159 mapread 0x00d5bf thru 0x00fa23 (0x2465 bytes) 000047 1788831564.072909 punch from 0x002399 to 0x00cada (0xa742 bytes) 000048 1788831564.078243 punch from 0x00ef87 to 0x00fa23 (0xa9d bytes) 000049 1788831564.088153 MIRROR_EXTEND to 2 mirrors 000050 1788831564.659122 mapwrite 0x01d455 thru 0x0276f2 (0xa29e bytes) 000053 1788831564.841458 trunc from 0x0276f3 to 0x00bad7 (0x1bc1b bytes) 000055 1788831564.864310 punch from 0x00637c to 0x00bad7 (0x575c bytes) 000056 1788831564.869669 mapwrite 0x0243f9 thru 0x025c6c (0x1874 bytes) 000057 1788831564.945289 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000058 1788831565.089798 punch from 0x01438e to 0x01441e (0x91 bytes) 000059 1788831565.098733 write 0x03127c thru 0x03e76e (0xd4f3 bytes) 000063 1788831565.106244 trunc from 0x03e76f to 0x03c61b (0x2153 bytes) 000064 1788831565.136685 punch from 0x015613 to 0x01efac (0x999a bytes) 000066 1788831565.161635 trunc from 0x03c61c to 0x0079c3 (0x34c58 bytes) 000069 1788831565.198095 mapread 0x005a5f thru 0x0079c3 (0x1f65 bytes) 000070 1788831565.240611 read 0x0015d1 thru 0x002ae0 (0x1510 bytes) 000071 1788831565.241655 read 0x0006dd thru 0x0079c3 (0x72e7 bytes) 000072 1788831565.252472 punch from 0x0054c5 to 0x0079c3 (0x24ff bytes) 000073 1788831565.258420 mapread 0x0064c2 thru 0x0069d4 (0x513 bytes) 000074 1788831565.266223 write 0x03e53b thru 0x03ffff (0x1ac5 bytes) 000075 1788831565.269139 write 0x03e486 thru 0x03ffff (0x1b7a bytes) 000077 1788831565.271515 trunc from 0x040000 to 0x01b864 (0x2479b bytes) 000081 1788831565.311103 MIRROR_EXTEND to 2 mirrors 000082 1788831565.964839 read 0x019e78 thru 0x01b864 (0x19ed bytes) 000083 1788831565.975860 read 0x019a7f thru 0x01b864 (0x1de6 bytes) 000084 1788831565.981120 punch from 0x00d79b to 0x019768 (0xbfce bytes) 000086 1788831566.068608 write 0x031936 thru 0x03ac54 (0x931f bytes) 000087 1788831566.069888 write 0x010474 thru 0x0120f8 (0x1c85 bytes) 000088 1788831566.085275 write 0x029784 thru 0x033b90 (0xa40d bytes) 000090 1788831566.088050 mapread 0x037e99 thru 0x03ac54 (0x2dbc bytes) 000091 1788831566.089661 mapwrite 0x01b7f6 thru 0x02598e (0xa199 bytes) 000093 1788831566.153806 trunc from 0x03ac55 to 0x01ddcb (0x1ce89 bytes) 000094 1788831566.198176 trunc from 0x01ddcc to 0x02ea9f (0x10cd4 bytes) 000095 1788831566.215581 mapread 0x02cf29 thru 0x02ea9f (0x1b77 bytes) 000096 1788831566.238196 MIRROR_RESYNC 2 mirrors 000097 1788831567.092016 read 0x02c6b7 thru 0x02c6ba (0x4 bytes) 000098 1788831567.110255 trunc from 0x02eaa0 to 0x00c02f (0x22a70 bytes) 000099 1788831567.261883 mapread 0x0060ca thru 0x00c02f (0x5f66 bytes) 000100 1788831567.302278 mapwrite 0x0072ce thru 0x015297 (0xdfca bytes) 000101 1788831567.412087 punch from 0x01336a to 0x015297 (0x1f2e bytes) 000103 1788831567.433212 mapread 0x010295 thru 0x015297 (0x5003 bytes) 000104 1788831567.445426 mapwrite 0x00dc80 thru 0x012a0b (0x4d8c bytes) 000105 1788831567.513371 read 0x00c5ad thru 0x015297 (0x8ceb bytes) 000106 1788831567.514013 punch from 0x009dd5 to 0x00fe6c (0x6098 bytes) 000109 1788831567.520016 read 0x00a88b thru 0x00b2b9 (0xa2f bytes) 000110 1788831567.529373 punch from 0x00169a to 0x00b49a (0x9e01 bytes) 000112 1788831567.538824 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000113 1788831567.683627 mapwrite 0x003448 thru 0x00ca50 (0x9609 bytes) 000114 1788831567.754816 mapwrite 0x006fae thru 0x0126e7 (0xb73a bytes) 000115 1788831567.815357 mapwrite 0x0069fc thru 0x015b2b (0xf130 bytes) 000116 1788831567.881785 trunc from 0x015b2c to 0x01101a (0x4b11 bytes) 000117 1788831567.899080 read 0x007704 thru 0x00c787 (0x5084 bytes) 000118 1788831567.900119 read 0x00fdb0 thru 0x01101a (0x126b bytes) 000120 1788831567.901208 mapwrite 0x00a4cd thru 0x00d696 (0x31ca bytes) 000125 1788831567.948873 mapwrite 0x01c88c thru 0x026c98 (0xa40d bytes) 000126 1788831568.029514 mapread 0x01b6d9 thru 0x026c98 (0xb5c0 bytes) 000128 1788831568.030940 mapread 0x01f157 thru 0x026c98 (0x7b42 bytes) 000129 1788831568.032984 write 0x03d11a thru 0x03ffff (0x2ee6 bytes) 000136 1788831568.039259 read 0x02defe thru 0x033efd (0x6000 bytes) 000137 1788831568.046820 write 0x02dcd0 thru 0x034ea1 (0x71d2 bytes) 000144 1788831568.051816 mapwrite 0x01945c thru 0x024c3f (0xb7e4 bytes) 000147 1788831568.127560 punch from 0x02a7d1 to 0x02fbc8 (0x53f8 bytes) 000150 1788831568.136861 write 0x0239d9 thru 0x028e29 (0x5451 bytes) 000151 1788831568.139602 read 0x01fa16 thru 0x020538 (0xb23 bytes) 000152 1788831568.140229 punch from 0x03509c to 0x03a232 (0x5197 bytes) 000153 1788831568.156267 trunc from 0x040000 to 0x022f2e (0x1d0d1 bytes) 000154 1788831568.176505 trunc from 0x022f2f to 0x033d18 (0x10dea bytes) 000155 1788831568.193454 mapread 0x013fca thru 0x01ca43 (0x8a7a bytes) 000159 1788831568.196289 punch from 0x0320ad to 0x033d18 (0x1c6c bytes) 000160 1788831568.202810 write 0x009f35 thru 0x0187b6 (0xe882 bytes) 000161 1788831568.204094 mapread 0x0124f6 thru 0x013a27 (0x1532 bytes) 000162 1788831568.204908 trunc from 0x033d19 to 0x02c485 (0x7893 bytes) 000163 1788831568.229980 write 0x00112b thru 0x01049e (0xf374 bytes) 000164 1788831568.232212 trunc from 0x02c486 to 0x00eca1 (0x1d7e4 bytes) 000165 1788831568.253683 read 0x006a77 thru 0x00eca1 (0x822b bytes) 000166 1788831568.254494 mapwrite 0x035ed2 thru 0x03e620 (0x874f bytes) 000167 1788831568.380918 MIRROR_EXTEND to 2 mirrors 000169 1788831569.136700 trunc from 0x03e621 to 0x03518a (0x9496 bytes) 000170 1788831569.258767 mapwrite 0x038c61 thru 0x03ffff (0x739f bytes) 000171 1788831569.402574 MIRROR_RESYNC 2 mirrors 000173 1788831570.285752 mapread 0x03c01e thru 0x03ffff (0x3fe2 bytes) 000174 1788831570.323095 write 0x01e719 thru 0x028c79 (0xa561 bytes) 000175 1788831570.427361 MIRROR_RESYNC 2 mirrors 000176 1788831571.217842 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000177 1788831571.369056 punch from 0x028d4a to 0x031209 (0x84c0 bytes) 000179 1788831571.385927 mapwrite 0x02fcde thru 0x034bbd (0x4ee0 bytes) 000180 1788831571.451761 mapread 0x02dd02 thru 0x032bfb (0x4efa bytes) 000181 1788831571.452606 mapread 0x034d27 thru 0x037780 (0x2a5a bytes) 000182 1788831571.467106 write 0x02d972 thru 0x031a10 (0x409f bytes) 000183 1788831571.468988 trunc from 0x040000 to 0x02b32f (0x14cd0 bytes) 000184 1788831571.486253 punch from 0x026224 to 0x02b32f (0x510c bytes) 000185 1788831571.495098 trunc from 0x02b330 to 0x02f5c9 (0x429a bytes) 000186 1788831571.514084 mapread 0x02d628 thru 0x02f5c9 (0x1fa2 bytes) 000188 1788831571.525352 write 0x0373ac thru 0x03f0a0 (0x7cf5 bytes) 000189 1788831571.527000 write 0x03e3e4 thru 0x03ffff (0x1c1c bytes) 000190 1788831571.529037 read 0x00cbd7 thru 0x01bd94 (0xf1be bytes) 000192 1788831571.540040 read 0x00cc52 thru 0x01075a (0x3b09 bytes) 000194 1788831571.544828 mapread 0x000e57 thru 0x00f611 (0xe7bb bytes) 000195 1788831571.566168 mapwrite 0x02700f thru 0x02f85a (0x884c bytes) 000197 1788831571.651929 mapread 0x03cd8c thru 0x03eb40 (0x1db5 bytes) 000198 1788831571.653684 mapread 0x035f1d thru 0x03ffff (0xa0e3 bytes) 000199 1788831571.654935 mapread 0x015309 thru 0x02421c (0xef14 bytes) 000200 1788831571.657831 punch from 0x030d7c to 0x03a85c (0x9ae1 bytes) 000202 1788831571.664337 punch from 0x03a2d4 to 0x03d583 (0x32b0 bytes) 000203 1788831571.671676 write 0x004f0b thru 0x0093f8 (0x44ee bytes) 000205 1788831571.675614 mapread 0x038b58 thru 0x03ffff (0x74a8 bytes) 000206 1788831571.707467 mapwrite 0x02a56a thru 0x02e91c (0x43b3 bytes) 000207 1788831571.765296 read 0x0383c9 thru 0x03a050 (0x1c88 bytes) 000208 1788831571.765838 trunc from 0x040000 to 0x021a52 (0x1e5ad bytes) 000209 1788831571.816010 mapread 0x00c368 thru 0x012f89 (0x6c22 bytes) 000211 1788831571.830847 mapwrite 0x024aa0 thru 0x029372 (0x48d3 bytes) 000212 1788831571.994335 trunc from 0x029373 to 0x02714e (0x2224 bytes) 000213 1788831572.027972 trunc from 0x02714f to 0x0166c1 (0x10a8d bytes) 000214 1788831572.058740 mapread 0x00b20e thru 0x0166c1 (0xb4b4 bytes) 000215 1788831572.061125 read 0x014a4d thru 0x0166c1 (0x1c75 bytes) 000221 1788831572.068072 trunc from 0x0166c2 to 0x01c5fc (0x5f3b bytes) 000223 1788831572.099433 read 0x004ed1 thru 0x00867b (0x37ab bytes) 000224 1788831572.101012 read 0x00bd4c thru 0x00d704 (0x19b9 bytes) 000225 1788831572.103035 mapread 0x0012eb thru 0x00550a (0x4220 bytes) 000226 1788831572.104334 trunc from 0x01c5fd to 0x034d06 (0x1870a bytes) 000228 1788831572.129050 read 0x014c07 thru 0x01f47a (0xa874 bytes) 000229 1788831572.150811 punch from 0x019b0f to 0x020b29 (0x701b bytes) 000238 1788831572.162790 mapread 0x02811b thru 0x034d06 (0xcbec bytes) 000240 1788831572.164370 trunc from 0x034d07 to 0x00db47 (0x271bf bytes) 000242 1788831572.186028 trunc from 0x00db48 to 0x01120a (0x36c3 bytes) 000244 1788831572.204313 read 0x002207 thru 0x006496 (0x4290 bytes) 000246 1788831572.206975 trunc from 0x01120b to 0x03dfcd (0x2cdc3 bytes) 000247 1788831572.228243 punch from 0x0292bf to 0x02e060 (0x4da2 bytes) 000248 1788831572.235019 read 0x0256bd thru 0x02dc62 (0x85a6 bytes) 000249 1788831572.247634 mapread 0x017453 thru 0x021328 (0x9ed6 bytes) 000250 1788831572.248981 trunc from 0x03dfce to 0x0151cf (0x28dfe bytes) 000251 1788831572.269562 read 0x0067dc thru 0x0071c8 (0x9ed bytes) 000253 1788831572.274287 read 0x003598 thru 0x009807 (0x6270 bytes) 000254 1788831572.276113 trunc from 0x0151d0 to 0x034d0c (0x1fb3d bytes) 000256 1788831572.299793 mapread 0x02a4dd thru 0x02b9dd (0x1501 bytes) 000257 1788831572.326475 read 0x03074f thru 0x0308e1 (0x193 bytes) 000258 1788831572.327029 trunc from 0x034d0d to 0x02574b (0xf5c1 bytes) 000260 1788831572.346905 mapread 0x00c36c thru 0x01606d (0x9d02 bytes) 000261 1788831572.348362 mapread 0x010652 thru 0x01bb58 (0xb507 bytes) 000264 1788831572.351816 trunc from 0x02574c to 0x038832 (0x130e7 bytes) 000265 1788831572.372553 mapread 0x02dfd0 thru 0x033b7e (0x5baf bytes) 000271 1788831572.388180 punch from 0x02a843 to 0x02f9e8 (0x51a6 bytes) 000274 1788831572.395803 mapwrite 0x029c72 thru 0x035023 (0xb3b2 bytes) 000276 1788831572.610472 mapread 0x03507e thru 0x03720e (0x2191 bytes) 000277 1788831572.616977 mapwrite 0x029913 thru 0x02cf77 (0x3665 bytes) 000279 1788831572.771649 punch from 0x011c30 to 0x0209cb (0xed9c bytes) 000281 1788831572.786853 mapread 0x0130b4 thru 0x021ed0 (0xee1d bytes) 000282 1788831572.923366 read 0x009fa5 thru 0x00a43c (0x498 bytes) 000283 1788831572.925092 mapwrite 0x0305fc thru 0x035fe4 (0x59e9 bytes) 000284 1788831572.986497 punch from 0x00b343 to 0x01b09a (0xfd58 bytes) 000285 1788831572.991546 punch from 0x01d73f to 0x022500 (0x4dc2 bytes) 000286 1788831572.996839 mapwrite 0x036383 thru 0x03ffff (0x9c7d bytes) 000287 1788831573.098824 write 0x011d3a thru 0x012573 (0x83a bytes) 000290 1788831573.102917 trunc from 0x040000 to 0x009695 (0x3696a bytes) 000294 1788831573.119976 read 0x00632a thru 0x009695 (0x336c bytes) 000296 1788831573.121732 punch from 0x004054 to 0x009695 (0x5642 bytes) 000297 1788831573.126150 write 0x0010c2 thru 0x00b504 (0xa443 bytes) 000298 1788831573.127225 mapread 0x002e82 thru 0x00b504 (0x8683 bytes) 000301 1788831573.129273 trunc from 0x00b505 to 0x000968 (0xab9c bytes) 000302 1788831573.146827 mapread 0x000689 thru 0x000968 (0x2e0 bytes) 000303 1788831573.157107 MIRROR_EXTEND to 2 mirrors 000305 1788831573.857776 trunc from 0x000969 to 0x02ac7a (0x2a312 bytes) 000306 1788831573.974259 punch from 0x010136 to 0x011cf3 (0x1bbe bytes) 000307 1788831573.983342 punch from 0x01ea42 to 0x0279f4 (0x8fb3 bytes) 000308 1788831573.994708 mapwrite 0x03a718 thru 0x03ffff (0x58e8 bytes) 000309 1788831574.197336 write 0x00ad73 thru 0x00cdf1 (0x207f bytes) 000310 1788831574.210429 mapread 0x002474 thru 0x00bba5 (0x9732 bytes) 000313 1788831574.235897 punch from 0x003e39 to 0x011e49 (0xe011 bytes) 000314 1788831574.250768 write 0x00e461 thru 0x014931 (0x64d1 bytes) 000316 1788831574.275672 trunc from 0x040000 to 0x023f57 (0x1c0a8 bytes) 000317 1788831574.295600 mapread 0x000c58 thru 0x002c75 (0x201e bytes) 000319 1788831574.312144 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x23f58): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23f58): Mirror 2: 0x99c0f13b (stale) CRC-32 checksum value for chunk [0, 0x23f58): Mirror 1: 0x584779b4 Mirror 3: 0x584779b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23f58: skipped 000320 1788831575.311232 write 0x03a2c7 thru 0x03ffff (0x5d39 bytes) 000323 1788831575.396987 trunc from 0x040000 to 0x0347ac (0xb853 bytes) 000325 1788831575.449344 trunc from 0x0347ad to 0x028fce (0xb7de bytes) 000326 1788831575.469336 trunc from 0x028fcf to 0x00ab94 (0x1e43a bytes) 000327 1788831575.485498 read 0x009e5d thru 0x00ab94 (0xd38 bytes) 000328 1788831575.490204 write 0x01badc thru 0x02ab21 (0xf046 bytes) 000329 1788831575.491788 write 0x008869 thru 0x00e9c5 (0x615d bytes) 000330 1788831575.499070 read 0x0293f5 thru 0x02ab21 (0x172d bytes) 000331 1788831575.499594 write 0x023439 thru 0x02b842 (0x840a bytes) 000332 1788831575.501266 punch from 0x01e6ce to 0x020edb (0x280e bytes) 000333 1788831575.521761 trunc from 0x02b843 to 0x01b634 (0x1020e bytes) 000334 1788831575.539745 trunc from 0x01b635 to 0x015500 (0x6134 bytes) 000335 1788831575.553587 write 0x0095cc thru 0x0140c5 (0xaafa bytes) 000336 1788831575.569405 write 0x021f2f thru 0x0250f1 (0x31c3 bytes) 000338 1788831575.571334 punch from 0x01ac2e to 0x020fed (0x63c0 bytes) 000339 1788831575.606797 write 0x009be4 thru 0x00a3ea (0x807 bytes) 000340 1788831575.607688 mapread 0x00f901 thru 0x00fda3 (0x4a3 bytes) 000342 1788831575.620997 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x250f2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x250f2): Mirror 2: 0xf772ffe1 (stale) Mirror 3: 0xddaea6e3 (stale) CRC-32 checksum value for chunk [0, 0x250f2): Mirror 1: 0x7b40eb2c Mirror 4: 0x7b40eb2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x250f2: skipped 000344 1788831576.868468 MIRROR_RESYNC 4 mirrors 000346 1788831577.651147 mapread 0x01e48d thru 0x0230c1 (0x4c35 bytes) 000347 1788831577.679066 punch from 0x0003bc to 0x003b96 (0x37db bytes) 000348 1788831577.787973 read 0x021ffd thru 0x0250f1 (0x30f5 bytes) 000349 1788831577.797343 mapread 0x01f029 thru 0x0250f1 (0x60c9 bytes) 000351 1788831577.810098 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x250f2): Mirror 3: 0x7b40eb2c (stale) Mirror 4: 0x7b40eb2c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x250f2): Mirror 3: 0x7b40eb2c (stale) Mirror 4: 0x7b40eb2c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x250f2: skipped 000352 1788831578.093295 mapwrite 0x02bf39 thru 0x03727e (0xb346 bytes) 000354 1788831578.209858 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3727f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3727f): Mirror 3: 0x1747a01f (stale) Mirror 4: 0x1747a01f (stale) CRC-32 checksum value for chunk [0, 0x3727f): Mirror 1: 0x7b386cb Mirror 5: 0x7b386cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3727f: skipped 000355 1788831578.946055 write 0x021846 thru 0x030b77 (0xf332 bytes) 000356 1788831579.078554 trunc from 0x03727f to 0x0134eb (0x23d93 bytes) 000357 1788831579.140265 punch from 0x006ab0 to 0x007fee (0x153f bytes) 000360 1788831579.145283 mapread 0x00cb60 thru 0x00fdfd (0x329e bytes) 000361 1788831579.175892 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x134ec): Mirror 3: 0x5401eabd (stale) Mirror 4: 0x5401eabd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x134ec): Mirror 3: 0x5401eabd (stale) Mirror 4: 0x5401eabd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x134ec: skipped 000363 1788831579.569699 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x134ec): Mirror 4: 0x5401eabd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x134ec): Mirror 4: 0x5401eabd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x134ec: skipped 000364 1788831579.875875 read 0x00d049 thru 0x00f93e (0x28f6 bytes) 000365 1788831579.882636 trunc from 0x0134ec to 0x0317ef (0x1e304 bytes) 000366 1788831579.903298 trunc from 0x0317f0 to 0x0121f1 (0x1f5fe bytes) 000367 1788831579.942113 write 0x03aacd thru 0x03ffff (0x5533 bytes) 000368 1788831579.944245 mapread 0x01b745 thru 0x01dd68 (0x2624 bytes) 000369 1788831579.988450 mapwrite 0x00d8d3 thru 0x0122fa (0x4a28 bytes) 000370 1788831580.112529 mapwrite 0x03607d thru 0x03ffff (0x9f83 bytes) 000373 1788831580.177659 write 0x02b19f thru 0x02ed61 (0x3bc3 bytes) 000374 1788831580.184230 mapread 0x01f461 thru 0x0229d4 (0x3574 bytes) 000375 1788831580.184975 write 0x0327c3 thru 0x0364f5 (0x3d33 bytes) 000376 1788831580.185947 punch from 0x01e5dc to 0x02a706 (0xc12b bytes) 000382 1788831580.198550 write 0x00560b thru 0x00d938 (0x832e bytes) 000383 1788831580.202196 trunc from 0x040000 to 0x01b420 (0x24bdf bytes) 000384 1788831580.217654 mapread 0x00e6d9 thru 0x014d36 (0x665e bytes) 000385 1788831580.219456 punch from 0x00811e to 0x0146d5 (0xc5b8 bytes) 000386 1788831580.244237 MIRROR_RESYNC 2 mirrors 000387 1788831580.967964 trunc from 0x01b421 to 0x0048d6 (0x16b4a bytes) 000389 1788831581.104491 trunc from 0x0048d7 to 0x013200 (0xe92a bytes) 000390 1788831581.127497 punch from 0x007457 to 0x00f66d (0x8217 bytes) 000391 1788831581.130615 mapwrite 0x001fff thru 0x003407 (0x1409 bytes) 000392 1788831581.213279 mapread 0x002107 thru 0x00298e (0x888 bytes) 000393 1788831581.214138 trunc from 0x013201 to 0x023453 (0x10253 bytes) 000394 1788831581.233830 trunc from 0x023454 to 0x02c4f1 (0x909e bytes) 000395 1788831581.278203 punch from 0x0041dd to 0x011e88 (0xdcac bytes) 000396 1788831581.287246 read 0x01ddac thru 0x0266e3 (0x8938 bytes) 000397 1788831581.304014 mapread 0x011172 thru 0x01582d (0x46bc bytes) 000398 1788831581.318038 trunc from 0x02c4f2 to 0x010e60 (0x1b691 bytes) 000402 1788831581.353839 trunc from 0x010e61 to 0x00d3a5 (0x3abb bytes) 000406 1788831581.380906 mapwrite 0x01a54b thru 0x026e0d (0xc8c3 bytes) 000407 1788831581.485980 write 0x03e18e thru 0x03ffff (0x1e72 bytes) 000408 1788831581.487968 read 0x02fec0 thru 0x03782b (0x796c bytes) 000410 1788831581.498171 read 0x016508 thru 0x016982 (0x47b bytes) 000411 1788831581.498718 read 0x039c72 thru 0x03ffff (0x638e bytes) 000413 1788831581.499941 mapwrite 0x0190ec thru 0x0204b8 (0x73cd bytes) 000415 1788831581.538966 read 0x02d5f5 thru 0x034598 (0x6fa4 bytes) 000416 1788831581.539688 mapread 0x00cfc0 thru 0x00d272 (0x2b3 bytes) 000417 1788831581.540980 trunc from 0x040000 to 0x039057 (0x6fa8 bytes) 000418 1788831581.580461 write 0x008a1c thru 0x013588 (0xab6d bytes) 000421 1788831581.585598 punch from 0x00fb89 to 0x012e64 (0x32dc bytes) 000422 1788831581.610536 read 0x001a65 thru 0x0057c4 (0x3d60 bytes) 000424 1788831581.616138 mapwrite 0x00c229 thru 0x00c73c (0x514 bytes) 000427 1788831581.710873 trunc from 0x039058 to 0x02cf59 (0xc0fe bytes) 000428 1788831581.743565 mapwrite 0x0108f9 thru 0x01e29e (0xd9a6 bytes) 000429 1788831581.829465 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000430 1788831582.002282 mapread 0x011bcb thru 0x01e8ad (0xcce3 bytes) 000432 1788831582.041947 mapread 0x00e6b6 thru 0x0148dc (0x6227 bytes) 000433 1788831582.049639 MIRROR_EXTEND to 2 mirrors 000434 1788831582.994362 punch from 0x025c92 to 0x02af06 (0x5275 bytes) 000437 1788831583.090939 read 0x006102 thru 0x014b69 (0xea68 bytes) 000440 1788831583.109803 mapwrite 0x0381a2 thru 0x03ffff (0x7e5e bytes) 000441 1788831583.224361 mapwrite 0x0209c9 thru 0x0273ee (0x6a26 bytes) 000442 1788831583.318374 MIRROR_RESYNC 2 mirrors 000443 1788831584.025947 mapread 0x013dbf thru 0x01d1a5 (0x93e7 bytes) 000444 1788831584.055934 mapwrite 0x00d485 thru 0x011c41 (0x47bd bytes) 000445 1788831584.414557 punch from 0x0203fa to 0x0211ee (0xdf5 bytes) 000447 1788831584.421312 read 0x00afef thru 0x0148e4 (0x98f6 bytes) 000448 1788831584.422563 read 0x03db5b thru 0x03ffff (0x24a5 bytes) 000449 1788831584.450719 read 0x00ee74 thru 0x015892 (0x6a1f bytes) 000450 1788831584.451206 mapread 0x02df52 thru 0x030445 (0x24f4 bytes) 000453 1788831584.456796 punch from 0x00482a to 0x013e86 (0xf65d bytes) 000455 1788831584.495751 MIRROR_RESYNC 2 mirrors 000456 1788831585.408210 mapread 0x018b89 thru 0x01bb1f (0x2f97 bytes) 000457 1788831585.429927 write 0x0034f5 thru 0x00c58d (0x9099 bytes) 000458 1788831585.516304 write 0x00ca33 thru 0x015e14 (0x93e2 bytes) 000459 1788831585.523795 trunc from 0x040000 to 0x016e0d (0x291f2 bytes) 000460 1788831585.557083 punch from 0x0099de to 0x00b874 (0x1e97 bytes) 000465 1788831585.575775 read 0x008015 thru 0x0114fd (0x94e9 bytes) 000466 1788831585.581260 write 0x025e81 thru 0x02aaa9 (0x4c29 bytes) 000467 1788831585.586729 MIRROR_RESYNC 2 mirrors 000468 1788831586.272136 mapwrite 0x008f73 thru 0x00b721 (0x27af bytes) 000469 1788831586.509961 mapread 0x0116de thru 0x012c6e (0x1591 bytes) 000473 1788831586.540164 write 0x008a7d thru 0x00bc21 (0x31a5 bytes) 000477 1788831586.544560 mapread 0x026b25 thru 0x02aaa9 (0x3f85 bytes) 000478 1788831586.583024 write 0x000636 thru 0x008336 (0x7d01 bytes) 000479 1788831586.594002 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000480 1788831586.843344 mapwrite 0x039f67 thru 0x03ffff (0x6099 bytes) 000481 1788831586.995055 trunc from 0x040000 to 0x005701 (0x3a8fe bytes) 000482 1788831587.011525 mapread 0x002ec0 thru 0x005701 (0x2842 bytes) 000483 1788831587.021327 write 0x018516 thru 0x01f6be (0x71a9 bytes) 000484 1788831587.026820 write 0x005622 thru 0x010460 (0xae3f bytes) 000485 1788831587.037303 punch from 0x01c29e to 0x01f6be (0x3421 bytes) 000487 1788831587.071901 trunc from 0x01f6bf to 0x015d79 (0x9945 bytes) 000488 1788831587.102172 mapwrite 0x024788 thru 0x02c584 (0x7dfd bytes) 000491 1788831587.192993 punch from 0x00df35 to 0x01b95b (0xda27 bytes) 000493 1788831587.212233 punch from 0x026e4b to 0x027e24 (0xfda bytes) 000494 1788831587.222827 read 0x02ac46 thru 0x02c584 (0x193f bytes) 000495 1788831587.223665 trunc from 0x02c585 to 0x033e6c (0x78e8 bytes) 000497 1788831587.248523 write 0x025478 thru 0x033081 (0xdc0a bytes) 000498 1788831587.260466 trunc from 0x033e6d to 0x009f23 (0x29f49 bytes) 000499 1788831587.299366 mapwrite 0x03e878 thru 0x03ffff (0x1788 bytes) 000501 1788831587.395191 write 0x033aca thru 0x03519f (0x16d6 bytes) 000502 1788831587.411096 mapwrite 0x03333a thru 0x03d028 (0x9cef bytes) 000504 1788831587.512558 write 0x02d2ca thru 0x038ddf (0xbb16 bytes) 000506 1788831587.523356 trunc from 0x040000 to 0x0395fc (0x6a03 bytes) 000507 1788831587.547158 mapread 0x00c82a thru 0x01c7ff (0xffd6 bytes) 000508 1788831587.574261 write 0x019a3c thru 0x01c61a (0x2bdf bytes) 000509 1788831587.575961 mapwrite 0x03c76d thru 0x03ffff (0x3893 bytes) 000510 1788831587.676294 trunc from 0x040000 to 0x02bc8a (0x14375 bytes) 000511 1788831587.703879 punch from 0x025d13 to 0x02bc8a (0x5f78 bytes) 000512 1788831587.711168 write 0x011762 thru 0x016bbb (0x545a bytes) 000513 1788831587.712325 mapwrite 0x00049c thru 0x001c46 (0x17ab bytes) 000514 1788831587.800205 read 0x029302 thru 0x02bc8a (0x2989 bytes) 000515 1788831587.818476 mapwrite 0x003a31 thru 0x006077 (0x2647 bytes) 000517 1788831587.879074 read 0x027824 thru 0x02bc8a (0x4467 bytes) 000518 1788831587.880874 mapread 0x01cd4d thru 0x02bc8a (0xef3e bytes) 000520 1788831587.883640 read 0x026174 thru 0x02b0cb (0x4f58 bytes) 000521 1788831587.884807 mapwrite 0x00c3b2 thru 0x01afe1 (0xec30 bytes) 000522 1788831587.956883 mapread 0x025abd thru 0x02bc8a (0x61ce bytes) 000523 1788831587.958301 punch from 0x01111c to 0x01ac8c (0x9b71 bytes) 000525 1788831587.965508 write 0x01deb0 thru 0x023a30 (0x5b81 bytes) 000526 1788831587.968041 mapwrite 0x00da18 thru 0x014b32 (0x711b bytes) 000527 1788831588.059298 punch from 0x0008b0 to 0x0093cf (0x8b20 bytes) 000528 1788831588.065607 write 0x0272a3 thru 0x036e7e (0xfbdc bytes) 000531 1788831588.070337 punch from 0x02ab97 to 0x0331cd (0x8637 bytes) 000533 1788831588.085796 trunc from 0x036e7f to 0x03c91e (0x5aa0 bytes) 000534 1788831588.107452 write 0x02190a thru 0x029bf9 (0x82f0 bytes) 000535 1788831588.109442 mapread 0x01fa01 thru 0x026268 (0x6868 bytes) 000538 1788831588.114112 write 0x006c85 thru 0x015c30 (0xefac bytes) 000539 1788831588.132111 mapwrite 0x00b2aa thru 0x00dcf0 (0x2a47 bytes) 000543 1788831588.215727 read 0x021a2c thru 0x02b095 (0x966a bytes) 000544 1788831588.228964 mapwrite 0x034482 thru 0x03ffff (0xbb7e bytes) 000545 1788831588.316918 read 0x0370ce thru 0x03ffff (0x8f32 bytes) 000547 1788831588.326166 MIRROR_EXTEND to 2 mirrors 000548 1788831589.223423 MIRROR_EXTEND to 3 mirrors 000549 1788831590.228993 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000551 1788831590.701995 punch from 0x0131ac to 0x01c4f0 (0x9345 bytes) 000552 1788831590.846112 MIRROR_RESYNC 2 mirrors 000553 1788831591.615098 read 0x00814c thru 0x016f2b (0xede0 bytes) 000555 1788831591.624723 mapread 0x019929 thru 0x0244c7 (0xab9f bytes) 000556 1788831591.678859 trunc from 0x040000 to 0x03e256 (0x1da9 bytes) 000558 1788831591.780044 mapwrite 0x028f8a thru 0x036c5c (0xdcd3 bytes) 000559 1788831591.917450 trunc from 0x03e257 to 0x0132ec (0x2af6a bytes) 000560 1788831591.940823 write 0x013987 thru 0x018a4f (0x50c9 bytes) 000561 1788831591.953051 read 0x002774 thru 0x00e5a4 (0xbe31 bytes) 000562 1788831591.961494 mapwrite 0x013d71 thru 0x0229c0 (0xec50 bytes) 000563 1788831592.043458 punch from 0x01dc7c to 0x01fd53 (0x20d8 bytes) 000564 1788831592.051549 read 0x008cf7 thru 0x016b9b (0xdea5 bytes) 000566 1788831592.052651 mapwrite 0x02400e thru 0x02e5dc (0xa5cf bytes) 000567 1788831592.134749 trunc from 0x02e5dd to 0x03e438 (0xfe5c bytes) 000568 1788831592.156982 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000569 1788831592.316880 mapwrite 0x035146 thru 0x03b08b (0x5f46 bytes) 000570 1788831592.476617 write 0x007d0b thru 0x016237 (0xe52d bytes) 000572 1788831592.485167 write 0x026e19 thru 0x0355a0 (0xe788 bytes) 000573 1788831592.491112 read 0x004e7c thru 0x00cc66 (0x7deb bytes) 000575 1788831592.497665 read 0x00f465 thru 0x019b7f (0xa71b bytes) 000576 1788831592.506807 write 0x02f52e thru 0x03e222 (0xecf5 bytes) 000578 1788831592.509653 mapwrite 0x00d126 thru 0x00f12d (0x2008 bytes) 000579 1788831592.577803 punch from 0x00451f to 0x005978 (0x145a bytes) 000581 1788831592.584412 mapwrite 0x00cced thru 0x0111ed (0x4501 bytes) 000582 1788831592.644824 read 0x0197f7 thru 0x025203 (0xba0d bytes) 000583 1788831592.645539 punch from 0x004fdb to 0x00e143 (0x9169 bytes) 000584 1788831592.651107 read 0x03d44b thru 0x03e438 (0xfee bytes) 000585 1788831592.652069 mapwrite 0x00f265 thru 0x018837 (0x95d3 bytes) 000589 1788831592.759914 MIRROR_EXTEND to 2 mirrors 000590 1788831593.826979 punch from 0x00f885 to 0x01f7e2 (0xff5e bytes) 000591 1788831593.948077 write 0x01cceb thru 0x02819b (0xb4b1 bytes) 000592 1788831593.979705 mapwrite 0x00cff6 thru 0x00ebb1 (0x1bbc bytes) 000593 1788831594.222033 MIRROR_RESYNC 2 mirrors 000594 1788831595.601241 mapread 0x03d632 thru 0x03e438 (0xe07 bytes) 000597 1788831595.630131 write 0x00cfdd thru 0x0196ce (0xc6f2 bytes) 000599 1788831595.736250 punch from 0x03420c to 0x03596a (0x175f bytes) 000600 1788831595.754032 write 0x03908c thru 0x03ffff (0x6f74 bytes) 000601 1788831595.776572 mapread 0x01a211 thru 0x029391 (0xf181 bytes) 000602 1788831595.810525 read 0x023cde thru 0x033452 (0xf775 bytes) 000604 1788831595.832669 mapread 0x03a160 thru 0x03ea6d (0x490e bytes) 000606 1788831595.850909 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000609 1788831596.102247 trunc from 0x040000 to 0x0304fa (0xfb05 bytes) 000611 1788831596.154777 mapwrite 0x03127d thru 0x0327c1 (0x1545 bytes) 000612 1788831596.275533 trunc from 0x0327c2 to 0x03ca7a (0xa2b9 bytes) 000617 1788831596.305587 punch from 0x014b0e to 0x018766 (0x3c59 bytes) 000618 1788831596.316348 write 0x039b69 thru 0x03ffff (0x6497 bytes) 000620 1788831596.338125 mapread 0x01b1d3 thru 0x01b3f3 (0x221 bytes) 000621 1788831596.365139 punch from 0x00f2e0 to 0x019987 (0xa6a8 bytes) 000622 1788831596.384656 write 0x0155d0 thru 0x01581a (0x24b bytes) 000626 1788831596.399252 trunc from 0x040000 to 0x016dd7 (0x29228 bytes) 000629 1788831596.422164 mapwrite 0x000bb6 thru 0x0104c8 (0xf913 bytes) 000631 1788831596.532270 mapread 0x00c7f8 thru 0x016dd7 (0xa5e0 bytes) 000632 1788831596.534774 punch from 0x004f8d to 0x00b299 (0x630d bytes) 000633 1788831596.540725 read 0x014d6d thru 0x016dd7 (0x206b bytes) 000636 1788831596.543138 punch from 0x008a37 to 0x01386f (0xae39 bytes) 000637 1788831596.549868 trunc from 0x016dd8 to 0x029480 (0x126a9 bytes) 000638 1788831596.574143 trunc from 0x029481 to 0x01c242 (0xd23e bytes) 000641 1788831596.616031 write 0x03feec thru 0x03ffff (0x114 bytes) 000644 1788831596.627246 mapwrite 0x018711 thru 0x01c9eb (0x42db bytes) 000645 1788831596.725449 write 0x02bb04 thru 0x03a03e (0xe53b bytes) 000646 1788831596.757844 read 0x03414c thru 0x035ae9 (0x199e bytes) 000648 1788831596.759229 trunc from 0x040000 to 0x021fe4 (0x1e01b bytes) 000649 1788831596.808134 mapread 0x009d3f thru 0x00f17e (0x5440 bytes) 000650 1788831596.823084 write 0x008fc7 thru 0x00a337 (0x1371 bytes) 000652 1788831596.825250 mapwrite 0x00ed91 thru 0x012735 (0x39a5 bytes) 000653 1788831596.895688 punch from 0x0007d9 to 0x002f20 (0x2748 bytes) 000655 1788831596.902249 mapread 0x017355 thru 0x01bf60 (0x4c0c bytes) 000656 1788831596.903507 punch from 0x0187ed to 0x021fe4 (0x97f8 bytes) 000659 1788831596.912203 mapread 0x01fd5e thru 0x021fe4 (0x2287 bytes) 000660 1788831596.929084 punch from 0x00be2d to 0x017311 (0xb4e5 bytes) 000662 1788831596.936118 write 0x016f30 thru 0x017253 (0x324 bytes) 000664 1788831596.945843 write 0x00e762 thru 0x012e4b (0x46ea bytes) 000665 1788831596.957368 trunc from 0x021fe5 to 0x00d20e (0x14dd6 bytes) 000667 1788831596.979207 mapread 0x00c6ed thru 0x00d20e (0xb22 bytes) 000668 1788831596.991309 write 0x00b130 thru 0x0105bd (0x548e bytes) 000669 1788831596.996068 mapwrite 0x0215da thru 0x0258bb (0x42e2 bytes) 000670 1788831597.046124 write 0x00eba0 thru 0x010da5 (0x2206 bytes) 000673 1788831597.053104 MIRROR_EXTEND to 2 mirrors 000674 1788831597.712269 write 0x02fc6e thru 0x03070f (0xaa2 bytes) 000675 1788831597.807261 write 0x0161d0 thru 0x024d5b (0xeb8c bytes) 000676 1788831597.820754 write 0x000835 thru 0x00a6e2 (0x9eae bytes) 000679 1788831597.833341 write 0x01c8d5 thru 0x029a2f (0xd15b bytes) 000680 1788831597.840370 punch from 0x004f2d to 0x00c2ad (0x7381 bytes) 000681 1788831597.863061 read 0x025e0c thru 0x03070f (0xa904 bytes) 000682 1788831597.873955 read 0x02de09 thru 0x03070f (0x2907 bytes) 000683 1788831597.874755 punch from 0x00e33c to 0x0130e1 (0x4da6 bytes) 000686 1788831597.881974 write 0x003f09 thru 0x007389 (0x3481 bytes) 000687 1788831597.889099 read 0x009513 thru 0x00c617 (0x3105 bytes) 000689 1788831597.899705 mapread 0x018c22 thru 0x0201ff (0x75de bytes) 000690 1788831597.901115 trunc from 0x030710 to 0x01148b (0x1f284 bytes) 000692 1788831597.959719 MIRROR_RESYNC 2 mirrors 000694 1788831598.668659 read 0x000e99 thru 0x008ce7 (0x7e4f bytes) 000695 1788831598.681195 punch from 0x000876 to 0x00ca61 (0xc1ec bytes) 000697 1788831598.758080 punch from 0x008dd7 to 0x00af59 (0x2183 bytes) 000699 1788831598.766071 mapread 0x00a513 thru 0x01148b (0x6f79 bytes) 000700 1788831598.791177 trunc from 0x01148c to 0x02afe1 (0x19b56 bytes) 000701 1788831598.818099 read 0x01fdc0 thru 0x025f54 (0x6195 bytes) 000703 1788831598.832943 trunc from 0x02afe2 to 0x005c22 (0x253bf bytes) 000704 1788831598.909960 punch from 0x000276 to 0x005c22 (0x59ad bytes) 000706 1788831598.919175 write 0x001b04 thru 0x00e62e (0xcb2b bytes) 000708 1788831598.945498 trunc from 0x00e62f to 0x0163f9 (0x7dcb bytes) 000709 1788831598.991162 mapread 0x00985e thru 0x00ee98 (0x563b bytes) 000710 1788831598.992064 write 0x01c9d3 thru 0x021e74 (0x54a2 bytes) 000711 1788831598.993358 write 0x015755 thru 0x01746e (0x1d1a bytes) 000712 1788831599.002749 write 0x02d579 thru 0x036894 (0x931c bytes) 000713 1788831599.004679 trunc from 0x036895 to 0x00e4b2 (0x283e2 bytes) 000714 1788831599.019131 read 0x000936 thru 0x002399 (0x1a64 bytes) 000715 1788831599.024806 punch from 0x00a9c0 to 0x00d5f2 (0x2c33 bytes) 000716 1788831599.041015 write 0x01ced0 thru 0x0289a1 (0xbad2 bytes) 000717 1788831599.043279 read 0x013678 thru 0x01b4c0 (0x7e49 bytes) 000718 1788831599.053196 punch from 0x00af2f to 0x0130ca (0x819c bytes) 000720 1788831599.065974 mapwrite 0x00b88e thru 0x00fa2b (0x419e bytes) 000721 1788831599.196261 write 0x0258ca thru 0x02e0d2 (0x8809 bytes) 000722 1788831599.197615 mapwrite 0x02417f thru 0x02fe7b (0xbcfd bytes) 000724 1788831599.284202 punch from 0x011a46 to 0x0217cb (0xfd86 bytes) 000725 1788831599.297153 mapwrite 0x03479a thru 0x038d48 (0x45af bytes) 000726 1788831599.419252 punch from 0x02fdda to 0x037e95 (0x80bc bytes) 000727 1788831599.429850 MIRROR_RESYNC 2 mirrors 000728 1788831600.342556 mapread 0x026f62 thru 0x0306a6 (0x9745 bytes) 000732 1788831600.381504 read 0x024786 thru 0x03417b (0xf9f6 bytes) 000733 1788831600.394949 read 0x010ca4 thru 0x016ab9 (0x5e16 bytes) 000734 1788831600.429332 trunc from 0x038d49 to 0x027210 (0x11b38 bytes) 000735 1788831600.528665 read 0x010b9b thru 0x018b1b (0x7f81 bytes) 000736 1788831600.542833 trunc from 0x027211 to 0x00aeeb (0x1c325 bytes) 000737 1788831600.568518 read 0x004d28 thru 0x00aeeb (0x61c4 bytes) 000738 1788831600.569281 write 0x01f472 thru 0x02d016 (0xdba5 bytes) 000740 1788831600.573814 read 0x01a923 thru 0x01f3d9 (0x4ab7 bytes) 000742 1788831600.583530 mapread 0x0240d0 thru 0x02b766 (0x7697 bytes) 000743 1788831600.584852 mapread 0x0223f0 thru 0x02d016 (0xac27 bytes) 000744 1788831600.590993 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d017): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d017): Mirror 2: 0x17383d0f (stale) CRC-32 checksum value for chunk [0, 0x2d017): Mirror 1: 0xf716046c Mirror 3: 0xf716046c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d017: skipped 000745 1788831601.700464 punch from 0x02643b to 0x02d016 (0x6bdc bytes) 000747 1788831601.871615 trunc from 0x02d017 to 0x00c213 (0x20e03 bytes) 000748 1788831601.918613 mapwrite 0x0118c1 thru 0x020a1b (0xf15b bytes) 000749 1788831602.017890 trunc from 0x020a1c to 0x005397 (0x1b684 bytes) 000750 1788831602.037094 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x5398): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5398): Mirror 2: 0x35d26e53 (stale) Mirror 3: 0x35d26e53 (stale) CRC-32 checksum value for chunk [0, 0x5398): Mirror 1: 0x35d26e53 Mirror 4: 0x35d26e53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5398: skipped 000751 1788831603.190128 read 0x000249 thru 0x005397 (0x514f bytes) 000752 1788831603.216960 read 0x0033c1 thru 0x005397 (0x1fd7 bytes) 000753 1788831603.223988 mapread 0x004c09 thru 0x005397 (0x78f bytes) 000754 1788831603.240830 punch from 0x00289e to 0x005397 (0x2afa bytes) 000755 1788831603.330555 read 0x00252a thru 0x005397 (0x2e6e bytes) 000757 1788831603.340208 mapread 0x002921 thru 0x005397 (0x2a77 bytes) 000758 1788831603.341506 punch from 0x003602 to 0x0037dd (0x1dc bytes) 000759 1788831603.349548 mapread 0x003871 thru 0x005397 (0x1b27 bytes) 000761 1788831603.360032 trunc from 0x005398 to 0x0065c8 (0x1231 bytes) 000762 1788831603.408578 mapwrite 0x038b4d thru 0x03ffff (0x74b3 bytes) 000763 1788831603.546949 trunc from 0x040000 to 0x011f4c (0x2e0b3 bytes) 000764 1788831603.567907 write 0x033673 thru 0x03ffff (0xc98d bytes) 000765 1788831603.569703 trunc from 0x040000 to 0x020ec0 (0x1f13f bytes) 000766 1788831603.594082 punch from 0x02039d to 0x020ec0 (0xb24 bytes) 000767 1788831603.599889 trunc from 0x020ec1 to 0x038f2a (0x1806a bytes) 000768 1788831603.622611 mapwrite 0x027933 thru 0x034355 (0xca23 bytes) 000770 1788831603.740151 trunc from 0x038f2b to 0x037554 (0x19d6 bytes) 000771 1788831603.760024 mapwrite 0x038f59 thru 0x03a6bd (0x1765 bytes) 000772 1788831603.865855 mapwrite 0x02af21 thru 0x02c96d (0x1a4d bytes) 000773 1788831603.914727 punch from 0x023024 to 0x031e64 (0xee41 bytes) 000774 1788831603.920876 mapread 0x012f6c thru 0x015016 (0x20ab bytes) 000775 1788831603.945798 write 0x0186e8 thru 0x01dbeb (0x5504 bytes) 000777 1788831603.959768 trunc from 0x03a6be to 0x03e121 (0x3a64 bytes) 000778 1788831603.993092 read 0x00b40a thru 0x0164ee (0xb0e5 bytes) 000779 1788831604.019349 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e122): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e122): Mirror 2: 0xb456e07c (stale) Mirror 3: 0xa61c09e5 (stale) Mirror 4: 0x274a30bf (stale) CRC-32 checksum value for chunk [0, 0x3e122): Mirror 1: 0x24e9c3cd Mirror 5: 0x24e9c3cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e122: skipped 000781 1788831605.126532 MIRROR_RESYNC 5 mirrors 000782 1788831606.373694 read 0x03d673 thru 0x03e121 (0xaaf bytes) 000783 1788831606.378602 read 0x00d813 thru 0x01aefe (0xd6ec bytes) 000784 1788831606.391906 MIRROR_RESYNC 5 mirrors 000787 1788831606.918955 write 0x02d148 thru 0x031921 (0x47da bytes) 000788 1788831607.017288 mapwrite 0x0173ec thru 0x0222e1 (0xaef6 bytes) 000789 1788831607.116955 punch from 0x03cada to 0x03e121 (0x1648 bytes) 000790 1788831607.126900 MIRROR_RESYNC 5 mirrors 000792 1788831608.848854 write 0x03f3af thru 0x03ffff (0xc51 bytes) 000794 1788831608.936776 punch from 0x034405 to 0x03e27d (0x9e79 bytes) 000796 1788831608.950845 mapwrite 0x02ba4b thru 0x03ab5f (0xf115 bytes) 000797 1788831609.035432 read 0x00c192 thru 0x01af44 (0xedb3 bytes) 000798 1788831609.049439 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x14ffc1b2 (stale) Mirror 3: 0x14ffc1b2 (stale) Mirror 4: 0x14ffc1b2 (stale) Mirror 5: 0x14ffc1b2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8fb645e9 Mirror 6: 0x8fb645e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000799 1788831609.989834 read 0x025ba7 thru 0x02b961 (0x5dbb bytes) 000801 1788831610.016888 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x14ffc1b2 (stale) Mirror 3: 0x14ffc1b2 (stale) Mirror 5: 0x14ffc1b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x14ffc1b2 (stale) Mirror 3: 0x14ffc1b2 (stale) Mirror 5: 0x14ffc1b2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8fb645e9 Mirror 6: 0x8fb645e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000802 1788831610.935358 read 0x0287fc thru 0x029a76 (0x127b bytes) 000804 1788831610.975977 punch from 0x000260 to 0x001f3a (0x1cdb bytes) 000805 1788831611.139702 mapread 0x00609b thru 0x015773 (0xf6d9 bytes) 000806 1788831611.278780 MIRROR_RESYNC 5 mirrors 000808 1788831612.483004 mapwrite 0x014f76 thru 0x0246ab (0xf736 bytes) 000809 1788831612.824746 punch from 0x03b1b0 to 0x03ffff (0x4e50 bytes) 000810 1788831612.836807 punch from 0x014913 to 0x019c34 (0x5322 bytes) 000814 1788831612.857246 MIRROR_RESYNC 5 mirrors 000816 1788831614.080863 write 0x00eb2b thru 0x0119ae (0x2e84 bytes) 000817 1788831614.179981 read 0x012cf7 thru 0x0192f1 (0x65fb bytes) 000819 1788831614.190060 read 0x0014a2 thru 0x01147d (0xffdc bytes) 000820 1788831614.202808 mapwrite 0x033dfa thru 0x0380b5 (0x42bc bytes) 000822 1788831614.265853 punch from 0x03aad0 to 0x03ffff (0x5530 bytes) 000823 1788831614.271946 mapwrite 0x01490c thru 0x01cbba (0x82af bytes) 000824 1788831614.326361 trunc from 0x040000 to 0x0137bb (0x2c844 bytes) 000825 1788831614.363093 MIRROR_RESYNC 5 mirrors 000826 1788831615.241004 mapwrite 0x019ef5 thru 0x01a5a7 (0x6b3 bytes) 000827 1788831615.416932 punch from 0x016111 to 0x01a5a7 (0x4497 bytes) 000829 1788831615.428238 trunc from 0x01a5a8 to 0x01f1d4 (0x4c2d bytes) 000830 1788831615.461743 read 0x007ff1 thru 0x00afeb (0x2ffb bytes) 000831 1788831615.469042 mapread 0x001b50 thru 0x007ebb (0x636c bytes) 000832 1788831615.483421 trunc from 0x01f1d5 to 0x02b53a (0xc366 bytes) 000834 1788831615.498184 trunc from 0x02b53b to 0x00511e (0x2641c bytes) 000835 1788831615.516350 mapwrite 0x0048f8 thru 0x006a7d (0x2186 bytes) 000836 1788831615.606258 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6a7e): Mirror 3: 0xbc72917a (stale) Mirror 5: 0xbc72917a (stale) Mirror 6: 0xbc72917a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6a7e): Mirror 3: 0xbc72917a (stale) Mirror 5: 0xbc72917a (stale) Mirror 6: 0xbc72917a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6a7e: skipped 000838 1788831616.050919 punch from 0x002642 to 0x006a7d (0x443c bytes) 000839 1788831616.063198 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6a7e): Mirror 5: 0xbc72917a (stale) Mirror 6: 0xbc72917a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6a7e): Mirror 5: 0xbc72917a (stale) Mirror 6: 0xbc72917a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6a7e: skipped 000840 1788831616.420040 mapwrite 0x032522 thru 0x036cee (0x47cd bytes) 000842 1788831616.611291 read 0x030c53 thru 0x035ef6 (0x52a4 bytes) 000843 1788831616.612012 write 0x017992 thru 0x01d40e (0x5a7d bytes) 000844 1788831616.639301 mapwrite 0x02183e thru 0x02be17 (0xa5da bytes) 000846 1788831616.729219 mapread 0x008650 thru 0x00e8cb (0x627c bytes) 000848 1788831616.745226 mapread 0x025276 thru 0x02f791 (0xa51c bytes) 000849 1788831616.746182 read 0x012886 thru 0x014ab7 (0x2232 bytes) 000850 1788831616.756295 mapread 0x02bbd8 thru 0x02f153 (0x357c bytes) 000851 1788831616.757763 mapread 0x014baa thru 0x022c68 (0xe0bf bytes) 000852 1788831616.758805 mapwrite 0x01d426 thru 0x02473c (0x7317 bytes) 000853 1788831616.815842 mapwrite 0x004116 thru 0x011d8f (0xdc7a bytes) 000854 1788831616.921667 punch from 0x02779e to 0x02fefb (0x875e bytes) 000857 1788831616.940601 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36cef): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36cef): Mirror 5: 0xcf5dc90d (stale) Mirror 6: 0xcf5dc90d (stale) CRC-32 checksum value for chunk [0, 0x36cef): Mirror 1: 0x6d6a6f2d Mirror 7: 0x6d6a6f2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36cef: skipped 000858 1788831618.010863 mapwrite 0x0349b3 thru 0x03feab (0xb4f9 bytes) 000861 1788831618.410191 read 0x023130 thru 0x023c26 (0xaf7 bytes) 000862 1788831618.427554 punch from 0x01a390 to 0x029306 (0xef77 bytes) 000863 1788831618.457937 read 0x02d8b3 thru 0x035cef (0x843d bytes) 000864 1788831618.490369 read 0x033d08 thru 0x034791 (0xa8a bytes) 000865 1788831618.501434 MIRROR_RESYNC 4 mirrors 000867 1788831620.031717 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000871 1788831620.739895 mapread 0x0273a2 thru 0x032ddb (0xba3a bytes) 000872 1788831620.794983 mapwrite 0x020843 thru 0x0234f5 (0x2cb3 bytes) 000873 1788831621.171347 punch from 0x03c18c to 0x03feab (0x3d20 bytes) 000875 1788831621.179509 mapwrite 0x00188f thru 0x00afa0 (0x9712 bytes) 000876 1788831621.241773 read 0x01e84b thru 0x02bcf7 (0xd4ad bytes) 000880 1788831621.260631 trunc from 0x03feac to 0x004b35 (0x3b376 bytes) 000881 1788831621.295659 write 0x035431 thru 0x03ffff (0xabcf bytes) 000882 1788831621.297124 punch from 0x017db3 to 0x01c7da (0x4a28 bytes) 000883 1788831621.314170 write 0x003a9d thru 0x005cea (0x224e bytes) 000885 1788831621.333182 write 0x029dcb thru 0x038aad (0xece3 bytes) 000886 1788831621.353241 write 0x01be64 thru 0x0216b0 (0x584d bytes) 000887 1788831621.378134 mapwrite 0x00e2d9 thru 0x017916 (0x963e bytes) 000888 1788831621.570558 mapread 0x00c8f1 thru 0x01b72a (0xee3a bytes) 000891 1788831621.576016 write 0x0115fa thru 0x01add8 (0x97df bytes) 000892 1788831621.582047 trunc from 0x040000 to 0x003ef4 (0x3c10b bytes) 000894 1788831621.626006 mapwrite 0x0273dd thru 0x02cf6d (0x5b91 bytes) 000896 1788831621.824306 punch from 0x008d2e to 0x00a2b0 (0x1583 bytes) 000897 1788831621.834297 write 0x0036b8 thru 0x0078aa (0x41f3 bytes) 000898 1788831621.866458 mapread 0x027ec6 thru 0x02ccb7 (0x4df2 bytes) 000899 1788831621.868786 trunc from 0x02cf6e to 0x03991c (0xc9af bytes) 000900 1788831621.909368 mapwrite 0x0243ef thru 0x026176 (0x1d88 bytes) 000901 1788831621.983979 trunc from 0x03991d to 0x03f91c (0x6000 bytes) 000902 1788831622.057165 mapwrite 0x03b827 thru 0x03ffff (0x47d9 bytes) 000903 1788831622.219513 punch from 0x030fcb to 0x035894 (0x48ca bytes) 000905 1788831622.236609 read 0x0358f7 thru 0x03761d (0x1d27 bytes) 000906 1788831622.246601 mapread 0x01b354 thru 0x01d2da (0x1f87 bytes) 000908 1788831622.252635 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xf2d17b44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xf2d17b44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000910 1788831622.636229 trunc from 0x040000 to 0x000ba3 (0x3f45c bytes) 000911 1788831622.661297 mapread 0x000509 thru 0x000ba3 (0x69b bytes) 000912 1788831622.679452 mapwrite 0x03d11d thru 0x03dca6 (0xb8a bytes) 000913 1788831622.902332 trunc from 0x03dca7 to 0x03ac65 (0x3041 bytes) 000914 1788831622.938292 read 0x020939 thru 0x028c9d (0x8365 bytes) 000915 1788831622.987993 mapread 0x01fc71 thru 0x026988 (0x6d18 bytes) 000917 1788831623.002915 trunc from 0x03ac66 to 0x029a7e (0x111e7 bytes) 000918 1788831623.062972 write 0x037d7d thru 0x03ffff (0x8283 bytes) 000919 1788831623.067073 mapwrite 0x008375 thru 0x01631d (0xdfa9 bytes) 000920 1788831623.259427 write 0x021525 thru 0x029f8f (0x8a6b bytes) 000921 1788831623.285495 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000922 1788831623.552002 mapwrite 0x009737 thru 0x017cf2 (0xe5bc bytes) 000924 1788831623.699919 write 0x026f8e thru 0x0369c2 (0xfa35 bytes) 000926 1788831623.717308 write 0x0248c4 thru 0x0287ae (0x3eeb bytes) 000927 1788831623.726068 mapwrite 0x0176e3 thru 0x01c789 (0x50a7 bytes) 000930 1788831623.808194 trunc from 0x040000 to 0x02cfaa (0x13055 bytes) 000931 1788831623.836548 punch from 0x002270 to 0x007441 (0x51d2 bytes) 000932 1788831623.841663 mapread 0x024fbb thru 0x02753a (0x2580 bytes) 000936 1788831623.849279 mapread 0x01bbd3 thru 0x0208e0 (0x4d0e bytes) 000938 1788831623.866773 read 0x01fe9c thru 0x02ace9 (0xae4e bytes) 000939 1788831623.867464 read 0x008e0b thru 0x00b79c (0x2992 bytes) 000940 1788831623.868001 punch from 0x022efe to 0x024a21 (0x1b24 bytes) 000942 1788831623.879297 punch from 0x0289ed to 0x02bb2b (0x313f bytes) 000943 1788831623.894918 trunc from 0x02cfab to 0x014e68 (0x18142 bytes) 000944 1788831623.942838 mapwrite 0x022276 thru 0x02eaf9 (0xc884 bytes) 000945 1788831624.140042 punch from 0x00cf93 to 0x0128aa (0x5918 bytes) 000946 1788831624.161628 mapwrite 0x02850a thru 0x035155 (0xcc4c bytes) 000947 1788831624.399679 trunc from 0x035156 to 0x00e3b9 (0x26d9c bytes) 000948 1788831624.424757 trunc from 0x00e3ba to 0x0135ef (0x5236 bytes) 000949 1788831624.450961 mapread 0x00c7fc thru 0x0135ef (0x6df4 bytes) 000952 1788831624.553523 mapread 0x013093 thru 0x0135ef (0x55d bytes) 000953 1788831624.554740 punch from 0x01310f to 0x0135ef (0x4e1 bytes) 000956 1788831624.587677 mapread 0x009ca7 thru 0x00c19f (0x24f9 bytes) 000957 1788831624.588588 mapread 0x011a4a thru 0x01251d (0xad4 bytes) 000958 1788831624.590622 mapwrite 0x01220b thru 0x01bf98 (0x9d8e bytes) 000960 1788831624.781628 mapwrite 0x0006e9 thru 0x004e1f (0x4737 bytes) 000961 1788831624.891559 write 0x039f6d thru 0x03ffff (0x6093 bytes) 000963 1788831624.895351 trunc from 0x040000 to 0x02c3f3 (0x13c0c bytes) 000965 1788831624.932492 read 0x0156d5 thru 0x020675 (0xafa1 bytes) 000967 1788831624.969870 mapread 0x0273b4 thru 0x02c3f3 (0x5040 bytes) 000968 1788831624.971410 read 0x017eac thru 0x027b68 (0xfcbd bytes) 000969 1788831624.972509 punch from 0x02acc4 to 0x02b857 (0xb94 bytes) 000974 1788831625.000593 read 0x01c8c7 thru 0x02c3f3 (0xfb2d bytes) 000975 1788831625.008176 trunc from 0x02c3f4 to 0x00a83a (0x21bb9 bytes) 000976 1788831625.024784 write 0x00679b thru 0x00ee0a (0x8670 bytes) 000982 1788831625.035006 write 0x038e9b thru 0x03fcf0 (0x6e56 bytes) 000983 1788831625.047256 punch from 0x02846c to 0x0302c9 (0x7e5e bytes) 000984 1788831625.066584 mapread 0x009352 thru 0x012499 (0x9148 bytes) 000985 1788831625.114744 read 0x03523e thru 0x03d4b6 (0x8279 bytes) 000986 1788831625.116625 mapwrite 0x01eaf2 thru 0x028be5 (0xa0f4 bytes) 000988 1788831625.192194 trunc from 0x03fcf1 to 0x01cd60 (0x22f90 bytes) 000989 1788831625.226628 punch from 0x012a6e to 0x01cd60 (0xa2f3 bytes) 000990 1788831625.231866 mapread 0x01a771 thru 0x01cd60 (0x25f0 bytes) 000991 1788831625.254284 read 0x011e8b thru 0x016b22 (0x4c98 bytes) 000992 1788831625.283033 trunc from 0x01cd61 to 0x01ed2c (0x1fcc bytes) 000993 1788831625.312798 trunc from 0x01ed2d to 0x0283da (0x96ae bytes) 000995 1788831625.327598 write 0x02dc84 thru 0x03b480 (0xd7fd bytes) 000996 1788831625.330047 trunc from 0x03b481 to 0x00f846 (0x2bc3a bytes) 000997 1788831625.345278 punch from 0x0083cb to 0x009b31 (0x1767 bytes) 000998 1788831625.353703 MIRROR_EXTEND to 2 mirrors 000999 1788831626.436487 MIRROR_EXTEND to 3 mirrors 001000 1788831627.650514 mapread 0x00c43a thru 0x00d0d6 (0xc9d bytes) All operations completed A-OK! PASS 70a (75s) == sanity-flr test 71: check mirror extend parent fid ==== 21:40:35 (1788831635) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 21:40:39 (1788831639) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16636 s, 12.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222061 s, 4.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 21:40:50 (1788831650) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1646144 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7039040 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3648544 .. Extending file size to 3668192 .. Extending file size to 6315680 .. Extending file size to 8184896 .. failed Extending file size to 8032384 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8039392 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8977568 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8242144 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8859776 .. Waiting 57375 57377 57378 57382 57383 failed 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:42:08 (1788831728) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6790112 .. Extending file size to 7217504 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6966016 .. Extending file size to 7219424 .. Extending file size to 8012064 .. Extending file size to 8486528 .. Extending file size to 9210048 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8654496 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9104864 .. Extending file size to 9244448 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 58861 58862 58863 58865 58866 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:43:33 (1788831813) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159031 s, 13.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.295068 s, 7.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa4d872df (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:43:56 (1788831836) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112127 s, 18.7 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 21:44:04 (1788831844) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d1:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:44:13 (1788831853) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.223096 s, 9.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:44:21 (1788831861) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.309919 s, 6.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:44:30 (1788831870) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.501963 s, 4.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:45:05 (1788831905) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103373 s, 10.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:46:09 (1788831969) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.64724 s, 3.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:47:13 (1788832033) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. Creating new pool test_204f oleg148-server: Pool lustre.test_204f created Adding targets to pool oleg148-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg148-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.384757 s, 5.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg148-server: Pool lustre.test_204f destroyed PASS 204f (87s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:48:40 (1788832120) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:48:47 (1788832127) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:48:55 (1788832135) oleg148-server: Pool lustre.test_206 created Adding targets to pool oleg148-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg148-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg148-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:49:21 (1788832161) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899103 s, 11.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460636 s, 22.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x69712753 Mirror 3: 0x69712753 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:49:33 (1788832173) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:49:37 (1788832177) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:49:41 (1788832181) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0942917 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ac0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1fb0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1fb0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1fb0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1290000 Read this (4 bytes): BBBB 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:50:18 (1788832218) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0284302 s, 288 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:50:26 (1788832226) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380279 s, 27.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f6 lmm_fid: [0x200000404:0x1f6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f6 lmm_fid: [0x200000404:0x1f6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834422 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 21:50:27.000000000 -0400 Modify: 2026-09-07 21:50:27.000000000 -0400 Change: 2026-09-07 21:50:29.000000000 -0400 Birth: 2026-09-07 21:50:28.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:50:34 (1788832234) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:50:38 (1788832238) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.736137 s, 55.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0544894 s, 75.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 7 21:50 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 7 21:50 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:50:47 (1788832247) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:50:56 (1788832256) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 1734 sec =========== 21:51:08 (1788832268) === sanity-flr: start cleanup 21:51:10 (1788832270) === === sanity-flr: finish cleanup 21:51:16 (1788832276) ===