-----============= acceptance-small: sanity-flr ============----- Tue Dec 17 05:10:27 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg203-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 05:10:38 (1734430238) === oleg203-client.virtnet: executing check_config_client /mnt/lustre oleg203-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg203-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6d0d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6d0d800.idle_timeout=debug disable quota as required oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 05:10:53 (1734430253) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f6980] == sanity-flr test 0a: lfs mirror create with -N option == 05:10:54 (1734430254) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:11:01 (1734430261) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:11:02 (1734430262) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 05:11:04 (1734430264) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:11:14 (1734430274) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:11:16 (1734430276) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 05:11:17 (1734430277) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6d0d800.stats=clear osc.lustre-OST0001-osc-ffff8800b6d0d800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:11:21 (1734430281) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:11:25 (1734430285) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:11:29 (1734430289) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 05:11:30 (1734430290) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 05:11:34 (1734430294) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:11:37 (1734430297) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:11:38 (1734430298) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:11:42 (1734430302) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 05:11:46 (1734430306) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:11:49 (1734430309) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 1.45748 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110717 s, 9.5 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:11:55 (1734430315) 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.157588 s, 106 MB/s PASS 22 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:12:00 (1734430320) Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6d0d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6d0d800.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6d0d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6d0d800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg203-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (31s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:12:32 (1734430352) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.23092 s, 40.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6d0d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6d0d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (20s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:12:52 (1734430372) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6d0d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6d0d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6d0d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6d0d800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg203-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (30s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:13:22 (1734430402) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.24018 s, 42.3 MB/s -rw-r--r-- 1 root root 52428800 Dec 17 05:13 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.536s user 0m0.002s sys 0m0.138s Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6d0d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6d0d800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.316s user 0m0.000s sys 0m0.139s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6d0d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6d0d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.406s user 0m0.000s sys 0m0.133s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg203-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg203-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6d0d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6d0d800.ost_server_uuid in FULL state after 0 sec PASS 33b (34s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:13:56 (1734430436) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:13:57 (1734430437) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:13:59 (1734430439) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 05:14:00 (1734430440) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 05:14:03 (1734430443) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.264895 s, 47.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:14:08 (1734430448) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115985 s, 90.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0238107 s, 44.0 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:14:12 (1734430452) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:14:15 (1734430455) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000679179 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00213232 s, 480 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00183817 s, 557 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00174195 s, 588 kB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 05:14:19 (1734430459) create test files with size 15731583 .. 852eb6731b8d0a943c4f60d077dc731c - 197823c44e6bd5affbda16b953aba91c - a0ef7d3763915b0bd0574b91f53704ad - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg203-client.virtnet /mnt/lustre (opts:) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre PASS 37 (10s) == sanity-flr test 38: resync ============================ 05:14:29 (1734430469) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:14:36 (1734430476) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:14:39 (1734430479) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0628394 s, 16.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0171589 s, 61.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 05:14:42 (1734430482) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0983762 s, 21.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.169256 s, 24.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.122141 s, 17.2 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 05:14:48 (1734430488) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 05:14:49 (1734430489) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:14:50 (1734430490) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108143 s, 97.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 05:14:52 (1734430492) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:14:53 (1734430493) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x13:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:14:56 (1734430496) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0721484 s, 145 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg203-client.virtnet /mnt/lustre (opts:) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (21s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:15:17 (1734430517) llite.lustre-ffff8800aa366000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0915208 s, 115 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg203-client.virtnet /mnt/lustre (opts:) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a1c21800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:15:21 (1734430521) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (12s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:15:33 (1734430533) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (12s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:15:45 (1734430545) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 05:15:46 (1734430546) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.100801 s, 31.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0881999 s, 35.7 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 334fc4ef4e3cf0902ac91dd55321ae21 - mirror 1 checksum: 7b1be6e715c5a687061a7b4b10568e4d - mirror 2 checksum: 334fc4ef4e3cf0902ac91dd55321ae21 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Dec 17 05:15:47 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Dec 17 05:15:47 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 334fc4ef4e3cf0902ac91dd55321ae21 - mirror 1 checksum: 7b1be6e715c5a687061a7b4b10568e4d - mirror 2 checksum: 7b1be6e715c5a687061a7b4b10568e4d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Dec 17 05:15:47 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Dec 17 05:15:47 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:15:51 (1734430551) Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 05:15:54 (1734430554) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 05:16:09 (1734430569) targets are mounted 05:16:09 (1734430569) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130255 s, 80.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (25s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:16:16 (1734430576) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:16:18 (1734430578) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:16:19 (1734430579) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00320971 s, 5.1 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:16:22 (1734430582) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:16:23 (1734430583) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:16:25 (1734430585) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (10s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:16:35 (1734430595) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:16:54 (1734430614) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:17:12 (1734430632) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:17:31 (1734430651) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 05:17:34 (1734430654) mirror create pid 22542 mirror split pid 22544 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 22542 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 22544 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 05:18:36 (1734430716) Seed set to 717 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1734430717.452463 write 0x015cbf thru 0x022e98 (0xd1da bytes) 000005 1734430717.455992 read 0x01b106 thru 0x022e98 (0x7d93 bytes) 000007 1734430717.456765 mapread 0x01b29d thru 0x022e98 (0x7bfc bytes) truncating to largest ever: 0xb82b 000008 1734430717.458312 trunc from 0x022e99 to 0x00b82a (0x1766e bytes) 000013 1734430717.473805 mapwrite 0x00b3be thru 0x011385 (0x5fc8 bytes) 000017 1734430717.575193 mapwrite 0x0386c0 thru 0x03a99e (0x22df bytes) truncating to largest ever: 0x14ef3 000018 1734430717.621288 trunc from 0x03a99f to 0x014ef2 (0x25aac bytes) 000019 1734430717.629462 mapread 0x014d7c thru 0x014ef2 (0x177 bytes) 000021 1734430717.634818 write 0x022ce6 thru 0x02716b (0x4486 bytes) truncating to largest ever: 0x30ffa 000023 1734430717.636067 trunc from 0x02716c to 0x030ff9 (0x9e8e bytes) 000025 1734430717.646831 read 0x00e13f thru 0x0121f7 (0x40b9 bytes) 000027 1734430717.647764 write 0x010afb thru 0x019e73 (0x9379 bytes) 000029 1734430717.654290 read 0x001f91 thru 0x00f19f (0xd20f bytes) truncating to largest ever: 0x3cf68 000030 1734430717.654862 trunc from 0x030ffa to 0x03cf67 (0xbf6e bytes) 000031 1734430717.661698 mapread 0x030329 thru 0x033b86 (0x385e bytes) 000032 1734430717.669872 mapread 0x001d38 thru 0x00d9c7 (0xbc90 bytes) 000033 1734430717.671306 mapwrite 0x035677 thru 0x03be18 (0x67a2 bytes) 000036 1734430717.720681 mapwrite 0x00fd42 thru 0x018cb7 (0x8f76 bytes) 000037 1734430717.756534 mapread 0x010e99 thru 0x01ba6c (0xabd4 bytes) 000041 1734430717.760900 write 0x01c59d thru 0x01ecce (0x2732 bytes) 000042 1734430717.761694 mapwrite 0x0219c2 thru 0x02cae1 (0xb120 bytes) 000043 1734430717.810334 trunc from 0x03cf68 to 0x0305e1 (0xc986 bytes) 000047 1734430717.819108 mapread 0x027a7e thru 0x029a67 (0x1fea bytes) 000049 1734430717.820457 mapwrite 0x00c9c1 thru 0x0194d2 (0xcb12 bytes) 000051 1734430717.880251 write 0x022c43 thru 0x023e61 (0x121f bytes) 000054 1734430717.881480 mapwrite 0x028be7 thru 0x03562c (0xca46 bytes) 000056 1734430717.953897 mapread 0x00fba3 thru 0x014e58 (0x52b6 bytes) 000057 1734430717.957287 MIRROR_EXTEND to 2 mirrors 000059 1734430718.208618 trunc from 0x03562d to 0x026217 (0xf415 bytes) 000061 1734430718.247855 read 0x01cc80 thru 0x026217 (0x9598 bytes) 000062 1734430718.257329 write 0x00a45b thru 0x013cc9 (0x986f bytes) 000063 1734430718.263515 trunc from 0x026218 to 0x03822e (0x12017 bytes) 000064 1734430718.271737 trunc from 0x03822f to 0x006159 (0x320d5 bytes) 000065 1734430718.282071 trunc from 0x00615a to 0x00f436 (0x92dd bytes) 000067 1734430718.290890 trunc from 0x00f437 to 0x00175d (0xdcd9 bytes) 000068 1734430718.299404 mapwrite 0x01be12 thru 0x01fe33 (0x4022 bytes) 000069 1734430718.351644 trunc from 0x01fe34 to 0x03720e (0x173db bytes) 000072 1734430718.360405 mapread 0x02b225 thru 0x02b27c (0x58 bytes) 000074 1734430718.366361 mapread 0x005ab7 thru 0x00f519 (0x9a63 bytes) 000075 1734430718.375351 read 0x02886a thru 0x034407 (0xbb9e bytes) 000079 1734430718.395099 trunc from 0x03720f to 0x0219d8 (0x15836 bytes) 000083 1734430718.404279 mapwrite 0x003941 thru 0x005359 (0x1a19 bytes) 000086 1734430718.437195 read 0x00d724 thru 0x012193 (0x4a70 bytes) 000087 1734430718.437720 mapwrite 0x03fef3 thru 0x03ffff (0x10d bytes) 000088 1734430718.489476 read 0x01dbfa thru 0x02458e (0x6995 bytes) 000090 1734430718.490779 mapwrite 0x00b9d7 thru 0x01a5da (0xec04 bytes) 000091 1734430718.530462 read 0x0073f4 thru 0x010817 (0x9424 bytes) 000092 1734430718.530962 read 0x008cc8 thru 0x017d3d (0xf076 bytes) 000093 1734430718.531379 mapread 0x0228cc thru 0x03122b (0xe960 bytes) 000094 1734430718.532636 read 0x00962d thru 0x00c5ac (0x2f80 bytes) 000096 1734430718.534744 read 0x03c179 thru 0x03ffff (0x3e87 bytes) 000098 1734430718.535879 mapread 0x03c593 thru 0x03ffff (0x3a6d bytes) 000100 1734430718.537468 write 0x033775 thru 0x036be9 (0x3475 bytes) 000101 1734430718.542776 MIRROR_EXTEND to 3 mirrors 000102 1734430718.806211 write 0x02a0f9 thru 0x02eb85 (0x4a8d bytes) 000106 1734430718.871936 read 0x01b9b8 thru 0x026280 (0xa8c9 bytes) 000110 1734430718.881478 trunc from 0x040000 to 0x014b1d (0x2b4e2 bytes) 000113 1734430718.913331 MIRROR_EXTEND to 4 mirrors 000116 1734430719.196802 read 0x0026bf thru 0x007f1f (0x5861 bytes) 000117 1734430719.205433 mapwrite 0x0193e3 thru 0x0292cf (0xfeed bytes) 000118 1734430719.307408 mapread 0x0008dd thru 0x00f69b (0xedbf bytes) 000120 1734430719.322255 mapread 0x01c485 thru 0x01fd22 (0x389e bytes) 000121 1734430719.322843 trunc from 0x0292d0 to 0x008064 (0x2126b bytes) 000122 1734430719.339313 MIRROR_EXTEND to 5 mirrors 000123 1734430719.634947 read 0x001160 thru 0x004325 (0x31c6 bytes) 000124 1734430719.645432 read 0x002702 thru 0x008064 (0x5963 bytes) 000125 1734430719.653335 mapwrite 0x01226e thru 0x01caaf (0xa842 bytes) 000126 1734430719.745346 mapwrite 0x0122f2 thru 0x01524c (0x2f5b bytes) 000128 1734430719.784700 MIRROR_RESYNC 5 mirrors 000129 1734430720.287601 read 0x01a70e thru 0x01c35e (0x1c51 bytes) 000133 1734430720.292947 trunc from 0x01cab0 to 0x004f45 (0x17b6a bytes) 000136 1734430720.335989 mapread 0x003cfe thru 0x004f45 (0x1248 bytes) 000138 1734430720.344885 MIRROR_RESYNC 5 mirrors 000139 1734430720.738947 read 0x0022c8 thru 0x002478 (0x1b1 bytes) 000143 1734430720.742909 trunc from 0x004f46 to 0x014aa8 (0xfb63 bytes) 000144 1734430720.786782 mapwrite 0x010dca thru 0x01c7a1 (0xb9d8 bytes) 000145 1734430720.852481 read 0x013379 thru 0x01c7a1 (0x9429 bytes) 000146 1734430720.853124 read 0x019e0f thru 0x01b9e3 (0x1bd5 bytes) 000147 1734430720.855990 MIRROR_RESYNC 5 mirrors 000148 1734430721.394458 write 0x036850 thru 0x03edff (0x85b0 bytes) 000150 1734430721.434508 mapwrite 0x009a90 thru 0x00b775 (0x1ce6 bytes) 000151 1734430721.503826 mapwrite 0x032c1a thru 0x036bc0 (0x3fa7 bytes) 000152 1734430721.550724 mapread 0x027b2c thru 0x02b13d (0x3612 bytes) 000153 1734430721.555176 trunc from 0x03ee00 to 0x027d56 (0x170a9 bytes) 000154 1734430721.562873 read 0x0245e2 thru 0x027d56 (0x3775 bytes) 000156 1734430721.563689 mapwrite 0x02b59e thru 0x037fb3 (0xca16 bytes) 000157 1734430721.635725 MIRROR_RESYNC 5 mirrors 000158 1734430722.149488 trunc from 0x037fb4 to 0x02d35e (0xac55 bytes) 000159 1734430722.226174 MIRROR_EXTEND to 6 mirrors 000161 1734430722.562085 write 0x01a5b4 thru 0x023aba (0x9507 bytes) 000162 1734430722.608904 trunc from 0x02d35f to 0x02bd2a (0x1634 bytes) 000169 1734430722.626516 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000172 1734430722.772953 write 0x03f119 thru 0x03ffff (0xee7 bytes) 000173 1734430722.774533 read 0x02240b thru 0x027ead (0x5aa3 bytes) 000174 1734430722.790060 MIRROR_RESYNC 5 mirrors 000178 1734430723.280898 read 0x013249 thru 0x02174e (0xe506 bytes) 000180 1734430723.286086 trunc from 0x040000 to 0x033f5c (0xc0a3 bytes) 000181 1734430723.322839 mapwrite 0x03466d thru 0x0363c8 (0x1d5c bytes) 000183 1734430723.375493 mapread 0x0037ac thru 0x009b0d (0x6362 bytes) 000184 1734430723.379820 read 0x029fed thru 0x034696 (0xa6aa bytes) 000185 1734430723.387896 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000186 1734430723.487682 read 0x031eb3 thru 0x0363c8 (0x4516 bytes) 000189 1734430723.493118 write 0x011f10 thru 0x01f7cb (0xd8bc bytes) 000190 1734430723.505441 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000191 1734430723.657251 mapwrite 0x031c64 thru 0x035052 (0x33ef bytes) 000192 1734430723.707413 mapread 0x013f76 thru 0x0201c7 (0xc252 bytes) 000194 1734430723.722140 read 0x009273 thru 0x00bc19 (0x29a7 bytes) 000197 1734430723.727785 read 0x023431 thru 0x02c287 (0x8e57 bytes) 000200 1734430723.733912 mapread 0x028b10 thru 0x02f829 (0x6d1a bytes) 000203 1734430723.737882 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000205 1734430723.822571 trunc from 0x0363c9 to 0x02d416 (0x8fb2 bytes) 000206 1734430723.832788 mapwrite 0x0238fd thru 0x029aa5 (0x61a9 bytes) 000211 1734430723.884060 read 0x007fcb thru 0x00b2c2 (0x32f8 bytes) 000213 1734430723.896500 write 0x009d18 thru 0x00c41b (0x2704 bytes) 000214 1734430723.901907 trunc from 0x02d417 to 0x027e4b (0x55cb bytes) 000215 1734430723.915638 read 0x0214f3 thru 0x023f2d (0x2a3b bytes) 000216 1734430723.916314 write 0x0333b8 thru 0x03ffff (0xcc48 bytes) 000218 1734430723.917854 mapwrite 0x000c2e thru 0x010766 (0xfb39 bytes) 000219 1734430723.968176 mapread 0x03d946 thru 0x03ffff (0x26ba bytes) 000220 1734430723.968790 mapread 0x009daf thru 0x00f220 (0x5472 bytes) 000222 1734430723.971345 mapwrite 0x00d058 thru 0x0113d3 (0x437c bytes) 000223 1734430724.014145 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000224 1734430724.078233 trunc from 0x040000 to 0x033386 (0xcc79 bytes) 000228 1734430724.086895 read 0x02b9b0 thru 0x02e2a8 (0x28f9 bytes) 000229 1734430724.092587 mapread 0x018f5e thru 0x01c070 (0x3113 bytes) 000230 1734430724.103844 mapread 0x01d7ea thru 0x02794f (0xa166 bytes) 000232 1734430724.113015 read 0x005496 thru 0x0103d4 (0xaf3f bytes) 000233 1734430724.119658 read 0x009a4f thru 0x011bdf (0x8191 bytes) 000234 1734430724.120082 trunc from 0x033387 to 0x004872 (0x2eb14 bytes) 000235 1734430724.132637 read 0x001013 thru 0x004872 (0x3860 bytes) 000237 1734430724.133486 write 0x01371a thru 0x021feb (0xe8d2 bytes) 000238 1734430724.134468 trunc from 0x021fec to 0x0364eb (0x14500 bytes) 000239 1734430724.144889 write 0x009fb1 thru 0x010a43 (0x6a93 bytes) 000241 1734430724.153751 trunc from 0x0364ec to 0x03a8f4 (0x4409 bytes) 000244 1734430724.166470 mapread 0x02521d thru 0x02e3c4 (0x91a8 bytes) truncating to largest ever: 0x3ea84 000245 1734430724.168171 trunc from 0x03a8f5 to 0x03ea83 (0x418f bytes) 000246 1734430724.175455 read 0x02da7b thru 0x031ca6 (0x422c bytes) 000252 1734430724.180152 mapread 0x029205 thru 0x02bb22 (0x291e bytes) 000254 1734430724.181902 mapread 0x00a401 thru 0x019b30 (0xf730 bytes) 000257 1734430724.184073 write 0x03b6d7 thru 0x03ffff (0x4929 bytes) 000259 1734430724.190358 write 0x023195 thru 0x0264f8 (0x3364 bytes) 000260 1734430724.190909 write 0x007aa7 thru 0x01271f (0xac79 bytes) 000261 1734430724.191557 trunc from 0x040000 to 0x01218d (0x2de72 bytes) 000262 1734430724.202125 read 0x000386 thru 0x00591b (0x5596 bytes) 000263 1734430724.202563 trunc from 0x01218e to 0x022c33 (0x10aa6 bytes) 000264 1734430724.211983 mapwrite 0x02666b thru 0x033e5e (0xd7f4 bytes) 000266 1734430724.265895 mapwrite 0x007cde thru 0x008196 (0x4b9 bytes) 000268 1734430724.297176 mapread 0x024310 thru 0x02de1e (0x9b0f bytes) 000271 1734430724.299303 trunc from 0x033e5f to 0x00c3a1 (0x27abd bytes) 000273 1734430724.310333 mapread 0x00b8ea thru 0x00c3a1 (0xab8 bytes) 000276 1734430724.311678 write 0x01f95a thru 0x029831 (0x9ed8 bytes) 000278 1734430724.312694 write 0x015b78 thru 0x01b222 (0x56ab bytes) 000279 1734430724.316734 write 0x01cfd4 thru 0x0218bd (0x48ea bytes) 000280 1734430724.323084 MIRROR_EXTEND to 2 mirrors 000281 1734430724.609371 trunc from 0x029832 to 0x01fd89 (0x9aa8 bytes) 000282 1734430724.675111 read 0x01a117 thru 0x01fd89 (0x5c73 bytes) 000283 1734430724.680176 read 0x001e79 thru 0x006727 (0x48af bytes) 000285 1734430724.686760 trunc from 0x01fd8a to 0x00776e (0x1861b bytes) 000287 1734430724.700853 write 0x0256ce thru 0x03041e (0xad51 bytes) 000288 1734430724.701907 trunc from 0x03041f to 0x02b7a6 (0x4c78 bytes) 000289 1734430724.715104 trunc from 0x02b7a7 to 0x019415 (0x12391 bytes) 000292 1734430724.729524 write 0x024d20 thru 0x02874c (0x3a2d bytes) 000293 1734430724.730324 read 0x00ed7f thru 0x019ba5 (0xae27 bytes) 000294 1734430724.741877 trunc from 0x02874d to 0x006ac8 (0x21c84 bytes) 000295 1734430724.752097 mapread 0x0019d9 thru 0x006ac8 (0x50f0 bytes) 000297 1734430724.753577 mapwrite 0x02bae0 thru 0x03ae1c (0xf33d bytes) 000298 1734430724.837230 MIRROR_EXTEND to 3 mirrors 000299 1734430725.103524 MIRROR_EXTEND to 4 mirrors 000302 1734430725.412555 trunc from 0x03ae1d to 0x00ec5f (0x2c1bd bytes) 000303 1734430725.477334 mapread 0x00ad84 thru 0x00ec5f (0x3edc bytes) 000304 1734430725.488523 mapwrite 0x028829 thru 0x0359aa (0xd182 bytes) 000305 1734430725.561362 MIRROR_RESYNC 4 mirrors 000306 1734430725.856122 MIRROR_RESYNC 4 mirrors 000307 1734430725.953693 read 0x00f892 thru 0x01f3e2 (0xfb51 bytes) 000308 1734430725.959860 trunc from 0x0359ab to 0x02102f (0x1497b bytes) 000309 1734430726.022445 read 0x01563a thru 0x01aed1 (0x5898 bytes) 000310 1734430726.029388 trunc from 0x021030 to 0x024a74 (0x3a45 bytes) 000312 1734430726.039402 write 0x035da7 thru 0x0368f4 (0xb4e bytes) 000313 1734430726.040068 mapread 0x010523 thru 0x01e603 (0xe0e1 bytes) 000316 1734430726.041115 trunc from 0x0368f5 to 0x0311fc (0x56f8 bytes) 000317 1734430726.047323 read 0x020362 thru 0x02e35b (0xdffa bytes) 000318 1734430726.051515 trunc from 0x0311fd to 0x01c61f (0x14bdd bytes) 000319 1734430726.065990 write 0x01202c thru 0x01245b (0x430 bytes) 000321 1734430726.067003 mapread 0x018277 thru 0x01b020 (0x2daa bytes) 000322 1734430726.067535 write 0x03d2f8 thru 0x03ffff (0x2d08 bytes) 000323 1734430726.068257 trunc from 0x040000 to 0x03b10e (0x4ef1 bytes) 000324 1734430726.079848 trunc from 0x03b10f to 0x01ed51 (0x1c3bd bytes) 000325 1734430726.088024 mapread 0x017334 thru 0x01ed51 (0x7a1e bytes) 000326 1734430726.098542 read 0x00d5b7 thru 0x0197ca (0xc214 bytes) 000327 1734430726.099040 write 0x037b9e thru 0x03ffff (0x8462 bytes) 000328 1734430726.099901 mapread 0x006a21 thru 0x00de48 (0x7428 bytes) 000329 1734430726.100496 write 0x00916a thru 0x014705 (0xb59c bytes) 000330 1734430726.101277 trunc from 0x040000 to 0x0129e4 (0x2d61b bytes) 000331 1734430726.108771 mapwrite 0x025774 thru 0x029175 (0x3a02 bytes) 000333 1734430726.168885 mapwrite 0x038666 thru 0x03ffff (0x799a bytes) 000335 1734430726.218854 mapread 0x034b35 thru 0x037ddd (0x32a9 bytes) 000336 1734430726.219626 write 0x01637a thru 0x01b48a (0x5111 bytes) 000337 1734430726.220230 mapread 0x01b0e8 thru 0x023f17 (0x8e30 bytes) 000338 1734430726.220709 trunc from 0x040000 to 0x01d219 (0x22de6 bytes) 000339 1734430726.228147 mapread 0x0101c5 thru 0x01d219 (0xd055 bytes) 000340 1734430726.229026 mapwrite 0x036c1a thru 0x03bfbd (0x53a4 bytes) 000341 1734430726.302367 write 0x008e14 thru 0x01272a (0x9917 bytes) 000342 1734430726.306700 MIRROR_RESYNC 4 mirrors 000343 1734430726.723431 trunc from 0x03bfbe to 0x00a9a7 (0x31616 bytes) 000344 1734430726.777197 mapwrite 0x030c1f thru 0x032722 (0x1b04 bytes) 000346 1734430726.865035 mapread 0x005596 thru 0x00b4fa (0x5f65 bytes) 000347 1734430726.875437 mapwrite 0x02513d thru 0x028662 (0x3526 bytes) 000352 1734430726.916949 mapread 0x016355 thru 0x0243b4 (0xe060 bytes) 000353 1734430726.925139 MIRROR_RESYNC 4 mirrors 000355 1734430727.375236 trunc from 0x032723 to 0x026739 (0xbfe9 bytes) 000356 1734430727.421216 MIRROR_EXTEND to 5 mirrors 000357 1734430727.707876 mapread 0x0171f1 thru 0x023b36 (0xc946 bytes) 000359 1734430727.738419 mapwrite 0x00ea5c thru 0x01163d (0x2be2 bytes) 000360 1734430727.875359 mapread 0x01cca9 thru 0x02445b (0x77b3 bytes) 000361 1734430727.888179 write 0x003fd4 thru 0x00cf82 (0x8faf bytes) 000362 1734430727.895180 read 0x02474d thru 0x026739 (0x1fed bytes) 000363 1734430727.901125 MIRROR_RESYNC 5 mirrors 000364 1734430728.398550 mapwrite 0x02acfe thru 0x038cac (0xdfaf bytes) 000365 1734430728.543796 write 0x007de2 thru 0x0120b5 (0xa2d4 bytes) 000366 1734430728.563222 mapread 0x01a9cc thru 0x024bb0 (0xa1e5 bytes) 000368 1734430728.581748 mapwrite 0x02d8bd thru 0x03cda6 (0xf4ea bytes) 000371 1734430728.640090 trunc from 0x03cda7 to 0x03376c (0x963a bytes) 000372 1734430728.652892 write 0x00b59b thru 0x013fe0 (0x8a46 bytes) 000374 1734430728.658507 mapread 0x029466 thru 0x0323c3 (0x8f5e bytes) 000376 1734430728.661323 trunc from 0x03376d to 0x000b0e (0x32c5e bytes) 000377 1734430728.680719 mapwrite 0x022059 thru 0x024404 (0x23ac bytes) 000378 1734430728.752742 trunc from 0x024405 to 0x030d4e (0xc94a bytes) 000379 1734430728.760666 mapread 0x00a9bb thru 0x01916b (0xe7b1 bytes) 000381 1734430728.773008 MIRROR_EXTEND to 6 mirrors 000383 1734430729.102434 mapread 0x020c9b thru 0x022a10 (0x1d76 bytes) 000385 1734430729.121573 trunc from 0x030d4f to 0x01df8a (0x12dc4 bytes) 000386 1734430729.195782 trunc from 0x01df8b to 0x0299a4 (0xba1a bytes) 000387 1734430729.213707 trunc from 0x0299a5 to 0x02c4c9 (0x2b25 bytes) 000388 1734430729.228747 MIRROR_EXTEND to 7 mirrors 000389 1734430729.560631 MIRROR_RESYNC 7 mirrors 000390 1734430730.191885 read 0x01e295 thru 0x02ae77 (0xcbe3 bytes) 000391 1734430730.200478 MIRROR_RESYNC 7 mirrors 000392 1734430730.329510 write 0x00e5cc thru 0x011c04 (0x3639 bytes) 000393 1734430730.373868 write 0x0050b4 thru 0x01357e (0xe4cb bytes) 000394 1734430730.386792 trunc from 0x02c4ca to 0x035a65 (0x959c bytes) 000395 1734430730.403531 write 0x01c4a6 thru 0x029017 (0xcb72 bytes) 000397 1734430730.413782 read 0x02c6f4 thru 0x02f514 (0x2e21 bytes) 000398 1734430730.420619 mapwrite 0x03de10 thru 0x03ffff (0x21f0 bytes) 000401 1734430730.496365 mapread 0x016a8f thru 0x01b27f (0x47f1 bytes) 000403 1734430730.497852 write 0x005585 thru 0x0116f9 (0xc175 bytes) 000406 1734430730.503566 mapwrite 0x009137 thru 0x00a283 (0x114d bytes) 000407 1734430730.538363 mapwrite 0x02a923 thru 0x02b2ba (0x998 bytes) 000408 1734430730.577389 mapwrite 0x01d8a7 thru 0x024b1f (0x7279 bytes) 000415 1734430730.634332 trunc from 0x040000 to 0x030678 (0xf987 bytes) 000421 1734430730.646469 read 0x00724e thru 0x0104a7 (0x925a bytes) 000424 1734430730.647366 mapwrite 0x01f095 thru 0x028180 (0x90ec bytes) 000426 1734430730.695457 mapwrite 0x0182b9 thru 0x018ef7 (0xc3f bytes) 000427 1734430730.728791 read 0x0010ad thru 0x00c5bd (0xb511 bytes) 000428 1734430730.729341 mapread 0x016098 thru 0x01c80a (0x6773 bytes) 000429 1734430730.729991 write 0x01a171 thru 0x01d597 (0x3427 bytes) 000430 1734430730.731237 mapread 0x02cb11 thru 0x030678 (0x3b68 bytes) 000432 1734430730.733061 trunc from 0x030679 to 0x0186f3 (0x17f85 bytes) 000434 1734430730.741370 read 0x001616 thru 0x00e7b6 (0xd1a1 bytes) 000435 1734430730.741840 trunc from 0x0186f4 to 0x0265bd (0xdeca bytes) 000436 1734430730.760804 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000437 1734430730.897233 read 0x006832 thru 0x010f3e (0xa70d bytes) 000438 1734430730.907504 read 0x01a59a thru 0x01b49d (0xf04 bytes) 000439 1734430730.916318 write 0x033a7f thru 0x03fa7f (0xc001 bytes) 000440 1734430730.918311 mapwrite 0x039b7b thru 0x03ffff (0x6485 bytes) 000442 1734430730.979469 mapread 0x039624 thru 0x03db0b (0x44e8 bytes) 000443 1734430730.982902 write 0x01c061 thru 0x028e7e (0xce1e bytes) 000446 1734430730.993101 read 0x005670 thru 0x010d92 (0xb723 bytes) 000447 1734430730.993558 mapread 0x023910 thru 0x02bf7f (0x8670 bytes) 000448 1734430730.994898 write 0x028511 thru 0x02c24b (0x3d3b bytes) 000450 1734430731.001948 MIRROR_EXTEND to 7 mirrors 000451 1734430731.386370 read 0x03ae32 thru 0x03ffff (0x51ce bytes) 000452 1734430731.390847 mapread 0x02ebbd thru 0x034629 (0x5a6d bytes) 000457 1734430731.403317 read 0x012908 thru 0x01db05 (0xb1fe bytes) 000458 1734430731.412080 trunc from 0x040000 to 0x020fa7 (0x1f058 bytes) 000461 1734430731.463666 read 0x019e3f thru 0x020fa7 (0x7169 bytes) 000463 1734430731.474278 mapread 0x0048e7 thru 0x007de7 (0x3501 bytes) 000464 1734430731.475769 mapread 0x003c52 thru 0x00b9a4 (0x7d53 bytes) 000465 1734430731.476691 mapread 0x0032b4 thru 0x00c434 (0x9181 bytes) 000466 1734430731.478010 write 0x037bc9 thru 0x03ef0b (0x7343 bytes) 000468 1734430731.484210 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000472 1734430731.642280 mapwrite 0x00185a thru 0x008e01 (0x75a8 bytes) 000475 1734430731.708194 write 0x000da4 thru 0x00fe57 (0xf0b4 bytes) 000476 1734430731.714015 read 0x018cfb thru 0x01aa28 (0x1d2e bytes) 000477 1734430731.720158 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000478 1734430731.851007 trunc from 0x03ef0c to 0x02a121 (0x14dea bytes) 000480 1734430731.863696 MIRROR_RESYNC 5 mirrors 000481 1734430732.482177 read 0x00ea37 thru 0x0171b0 (0x877a bytes) 000482 1734430732.491482 mapwrite 0x01ec98 thru 0x0201aa (0x1513 bytes) 000484 1734430732.633631 mapwrite 0x02ae05 thru 0x02e45f (0x365b bytes) 000488 1734430732.701167 read 0x027c0e thru 0x02e45f (0x6852 bytes) 000492 1734430732.702478 mapwrite 0x03714c thru 0x039db9 (0x2c6e bytes) 000494 1734430732.768325 write 0x018c0d thru 0x01b122 (0x2516 bytes) 000495 1734430732.769055 read 0x037586 thru 0x039db9 (0x2834 bytes) 000496 1734430732.769448 read 0x0053d1 thru 0x00f704 (0xa334 bytes) truncating to largest ever: 0x3edfd 000498 1734430732.770265 trunc from 0x039dba to 0x03edfc (0x5043 bytes) 000499 1734430732.790083 mapwrite 0x02b092 thru 0x02f60c (0x457b bytes) 000500 1734430732.836657 trunc from 0x03edfd to 0x02b6ef (0x1370d bytes) 000501 1734430732.865461 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000503 1734430733.006848 read 0x027791 thru 0x02b6ef (0x3f5f bytes) 000504 1734430733.013409 read 0x020231 thru 0x02b6ef (0xb4bf bytes) 000506 1734430733.020398 read 0x012192 thru 0x018f1d (0x6d8c bytes) 000508 1734430733.021176 write 0x00c0cb thru 0x0112fe (0x5234 bytes) 000509 1734430733.021892 trunc from 0x02b6f0 to 0x010e4e (0x1a8a1 bytes) 000510 1734430733.032827 read 0x008ffe thru 0x010e4e (0x7e51 bytes) 000511 1734430733.033322 mapwrite 0x02faa0 thru 0x03150d (0x1a6e bytes) 000512 1734430733.105813 mapread 0x007ef2 thru 0x011da0 (0x9eaf bytes) 000513 1734430733.108648 mapwrite 0x00d666 thru 0x00f3d2 (0x1d6d bytes) 000514 1734430733.143544 write 0x03a9b0 thru 0x03ffff (0x5650 bytes) 000515 1734430733.146326 write 0x01bd72 thru 0x020ca5 (0x4f34 bytes) 000517 1734430733.156560 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000518 1734430733.371454 mapread 0x02c68d thru 0x02dda5 (0x1719 bytes) 000519 1734430733.385384 trunc from 0x040000 to 0x013827 (0x2c7d8 bytes) 000521 1734430733.411553 read 0x009a99 thru 0x00c1b2 (0x271a bytes) 000522 1734430733.427407 mapread 0x00c5ea thru 0x013827 (0x723e bytes) 000524 1734430733.439405 mapwrite 0x027076 thru 0x036d8a (0xfd15 bytes) 000526 1734430733.536603 mapwrite 0x038c06 thru 0x03ffff (0x73fa bytes) 000527 1734430733.597437 read 0x0110be thru 0x020ba2 (0xfae5 bytes) 000528 1734430733.605778 trunc from 0x040000 to 0x01190f (0x2e6f0 bytes) 000530 1734430733.615005 trunc from 0x011910 to 0x030c30 (0x1f321 bytes) 000532 1734430733.626793 trunc from 0x030c31 to 0x00ec5f (0x21fd1 bytes) 000535 1734430733.638875 mapwrite 0x01b001 thru 0x0269fe (0xb9fe bytes) 000539 1734430733.736582 mapread 0x01f674 thru 0x0269fe (0x738b bytes) 000546 1734430733.740030 mapread 0x022f59 thru 0x0269fe (0x3aa6 bytes) 000547 1734430733.740645 mapwrite 0x01998d thru 0x01f881 (0x5ef5 bytes) 000548 1734430733.797466 MIRROR_RESYNC 3 mirrors 000550 1734430734.179681 read 0x0264de thru 0x0269fe (0x521 bytes) 000551 1734430734.180763 read 0x01f6ce thru 0x021cea (0x261d bytes) 000553 1734430734.187817 mapwrite 0x010d87 thru 0x01d8da (0xcb54 bytes) 000554 1734430734.338298 mapread 0x01d07c thru 0x0269fe (0x9983 bytes) 000555 1734430734.366210 write 0x038ec9 thru 0x03cd4e (0x3e86 bytes) 000556 1734430734.367314 read 0x024c48 thru 0x026b89 (0x1f42 bytes) 000557 1734430734.368288 mapread 0x023414 thru 0x02d628 (0xa215 bytes) 000558 1734430734.377539 mapread 0x02b727 thru 0x039440 (0xdd1a bytes) 000559 1734430734.378167 mapwrite 0x00ae38 thru 0x018a45 (0xdc0e bytes) 000560 1734430734.425747 trunc from 0x03cd4f to 0x01f345 (0x1da09 bytes) 000562 1734430734.441162 trunc from 0x01f346 to 0x01092b (0xea1a bytes) 000563 1734430734.455989 mapread 0x00a7fe thru 0x0101b1 (0x59b4 bytes) 000567 1734430734.459255 mapread 0x010412 thru 0x01057d (0x16c bytes) 000568 1734430734.460269 trunc from 0x01092c to 0x007b05 (0x8e26 bytes) 000571 1734430734.472592 write 0x006a09 thru 0x010dc2 (0xa3ba bytes) 000573 1734430734.476742 MIRROR_EXTEND to 4 mirrors 000574 1734430734.759869 mapread 0x00e41a thru 0x010dc2 (0x29a9 bytes) 000575 1734430734.774463 trunc from 0x010dc3 to 0x014239 (0x3477 bytes) 000577 1734430734.817695 MIRROR_EXTEND to 5 mirrors 000578 1734430735.103199 MIRROR_RESYNC 5 mirrors 000580 1734430735.512961 MIRROR_EXTEND to 6 mirrors 000581 1734430735.833751 mapwrite 0x03210c thru 0x03ffff (0xdef4 bytes) 000582 1734430735.972610 write 0x023683 thru 0x0298ac (0x622a bytes) 000583 1734430735.980756 write 0x029ba9 thru 0x03704e (0xd4a6 bytes) 000584 1734430735.982678 mapread 0x03aea7 thru 0x03ffff (0x5159 bytes) 000588 1734430735.984323 mapwrite 0x0204db thru 0x0269f4 (0x651a bytes) 000590 1734430736.034813 MIRROR_EXTEND to 7 mirrors 000593 1734430736.362710 read 0x03f0b5 thru 0x03ffff (0xf4b bytes) 000594 1734430736.367952 trunc from 0x040000 to 0x034019 (0xbfe6 bytes) 000597 1734430736.429828 write 0x02ab05 thru 0x030997 (0x5e93 bytes) 000598 1734430736.440143 mapwrite 0x019474 thru 0x01a40a (0xf97 bytes) 000599 1734430736.503725 mapread 0x024bd9 thru 0x028cdc (0x4104 bytes) 000601 1734430736.510680 mapwrite 0x018214 thru 0x021eb8 (0x9ca5 bytes) 000602 1734430736.550196 read 0x01eec5 thru 0x0275d9 (0x8715 bytes) 000603 1734430736.550819 trunc from 0x03401a to 0x029f48 (0xa0d1 bytes) 000604 1734430736.564154 mapwrite 0x030cf2 thru 0x032da3 (0x20b2 bytes) 000606 1734430736.620948 mapread 0x002f88 thru 0x00f087 (0xc100 bytes) 000610 1734430736.623960 write 0x034b8b thru 0x03b1ff (0x6675 bytes) 000612 1734430736.627295 mapread 0x015a6f thru 0x0244a5 (0xea37 bytes) 000613 1734430736.630509 write 0x0203bf thru 0x02cc65 (0xc8a7 bytes) 000615 1734430736.633252 read 0x008626 thru 0x0167d8 (0xe1b3 bytes) 000617 1734430736.634593 trunc from 0x03b200 to 0x0308aa (0xa955 bytes) 000620 1734430736.651995 mapread 0x004db3 thru 0x0149cd (0xfc1b bytes) 000624 1734430736.654076 trunc from 0x0308ab to 0x03d041 (0xc797 bytes) 000625 1734430736.670940 MIRROR_EXTEND to 8 mirrors 000626 1734430737.006736 mapwrite 0x00c89f thru 0x014528 (0x7c8a bytes) 000627 1734430737.139329 mapwrite 0x000350 thru 0x000fbd (0xc6e bytes) 000629 1734430737.178470 write 0x02a108 thru 0x037869 (0xd762 bytes) 000630 1734430737.188508 write 0x0167a6 thru 0x024e37 (0xe692 bytes) 000631 1734430737.196615 mapread 0x037051 thru 0x03c158 (0x5108 bytes) 000633 1734430737.204596 mapread 0x021556 thru 0x02eaf5 (0xd5a0 bytes) 000634 1734430737.205524 trunc from 0x03d042 to 0x006776 (0x368cb bytes) 000635 1734430737.235289 write 0x009217 thru 0x00f835 (0x661f bytes) 000638 1734430737.236843 read 0x0036b9 thru 0x005fdc (0x2924 bytes) truncating to largest ever: 0x3fb18 000639 1734430737.237289 trunc from 0x00f836 to 0x03fb17 (0x302e2 bytes) 000641 1734430737.255460 write 0x03006f thru 0x0310d7 (0x1069 bytes) 000642 1734430737.262842 mapread 0x022e2a thru 0x023917 (0xaee bytes) 000643 1734430737.269330 mapread 0x025c4c thru 0x02fa39 (0x9dee bytes) 000644 1734430737.269957 mapread 0x02fd4a thru 0x03d40f (0xd6c6 bytes) 000647 1734430737.275887 trunc from 0x03fb18 to 0x002f84 (0x3cb93 bytes) 000649 1734430737.288529 write 0x03bf58 thru 0x03fdad (0x3e56 bytes) 000651 1734430737.293306 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000652 1734430737.424078 write 0x00c9a7 thru 0x01a2a6 (0xd900 bytes) 000654 1734430737.439340 MIRROR_RESYNC 7 mirrors 000656 1734430738.025347 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000657 1734430738.187992 read 0x0361a5 thru 0x03fba6 (0x9a02 bytes) 000659 1734430738.194812 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000661 1734430738.335699 write 0x032bf9 thru 0x033f64 (0x136c bytes) 000662 1734430738.407645 write 0x00dace thru 0x01cd42 (0xf275 bytes) 000663 1734430738.424643 read 0x02f23f thru 0x03a1d8 (0xaf9a bytes) 000664 1734430738.431583 read 0x039f7c thru 0x03fdad (0x5e32 bytes) 000671 1734430738.443956 MIRROR_EXTEND to 6 mirrors 000675 1734430738.763894 mapread 0x015a5f thru 0x01c1a2 (0x6744 bytes) 000676 1734430738.785386 mapwrite 0x010b4f thru 0x015cca (0x517c bytes) 000677 1734430738.888890 mapwrite 0x00e676 thru 0x0167bc (0x8147 bytes) 000680 1734430738.920865 mapwrite 0x0045b5 thru 0x01424f (0xfc9b bytes) 000681 1734430738.971413 write 0x010d8b thru 0x01188a (0xb00 bytes) 000683 1734430738.972339 trunc from 0x03fdae to 0x00663a (0x39773 bytes) 000684 1734430738.990578 trunc from 0x00663b to 0x01d00a (0x169d0 bytes) 000685 1734430738.998761 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000687 1734430739.106272 mapwrite 0x0223fb thru 0x02c6d7 (0xa2dd bytes) 000693 1734430739.188046 write 0x009104 thru 0x00cef7 (0x3df4 bytes) 000694 1734430739.195692 mapread 0x017e0f thru 0x0271fc (0xf3ee bytes) 000695 1734430739.211796 MIRROR_EXTEND to 6 mirrors 000696 1734430739.452441 read 0x01ddfd thru 0x0273bd (0x95c1 bytes) 000697 1734430739.458746 trunc from 0x02c6d8 to 0x00655c (0x2617b bytes) 000698 1734430739.519169 write 0x02de1f thru 0x03c9d8 (0xebba bytes) 000700 1734430739.521906 trunc from 0x03c9d9 to 0x006454 (0x36584 bytes) 000701 1734430739.535214 mapread 0x001149 thru 0x006454 (0x530c bytes) 000702 1734430739.543418 mapread 0x0012f4 thru 0x002274 (0xf81 bytes) 000707 1734430739.544743 write 0x01155b thru 0x01c0a1 (0xab47 bytes) 000715 1734430739.548297 read 0x0096fd thru 0x010c58 (0x755c bytes) 000719 1734430739.559443 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000720 1734430739.658739 mapwrite 0x039b0a thru 0x03eebf (0x53b6 bytes) 000723 1734430739.727196 mapwrite 0x02292f thru 0x031c6d (0xf33f bytes) 000724 1734430739.787490 write 0x0316b1 thru 0x03db8b (0xc4db bytes) 000725 1734430739.793442 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000726 1734430739.932682 mapread 0x0282c0 thru 0x030325 (0x8066 bytes) 000727 1734430739.949386 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000728 1734430740.039169 read 0x038389 thru 0x03eebf (0x6b37 bytes) 000730 1734430740.044767 read 0x002225 thru 0x004a3f (0x281b bytes) 000731 1734430740.064900 MIRROR_RESYNC 3 mirrors 000734 1734430740.411988 trunc from 0x03eec0 to 0x025feb (0x18ed4 bytes) 000735 1734430740.451119 read 0x011aff thru 0x016870 (0x4d72 bytes) 000736 1734430740.457119 write 0x002cf8 thru 0x00bf09 (0x9212 bytes) 000737 1734430740.462656 trunc from 0x025fec to 0x01c360 (0x9c8b bytes) 000738 1734430740.476850 write 0x02245a thru 0x02c84e (0xa3f5 bytes) 000739 1734430740.478687 mapread 0x015f5d thru 0x022b69 (0xcc0d bytes) 000740 1734430740.485817 read 0x01775c thru 0x019fad (0x2852 bytes) 000741 1734430740.486232 read 0x002119 thru 0x006610 (0x44f8 bytes) 000742 1734430740.487179 write 0x011147 thru 0x01d206 (0xc0c0 bytes) 000743 1734430740.488019 read 0x0124c2 thru 0x014a43 (0x2582 bytes) 000744 1734430740.488359 read 0x02a8ec thru 0x02b3bf (0xad4 bytes) 000746 1734430740.488977 trunc from 0x02c84f to 0x03154e (0x4d00 bytes) 000747 1734430740.501151 mapwrite 0x00e1c0 thru 0x01a27b (0xc0bc bytes) 000751 1734430740.562840 trunc from 0x03154f to 0x001535 (0x30019 bytes) 000756 1734430740.579709 mapread 0x001220 thru 0x001535 (0x316 bytes) 000759 1734430740.581224 mapwrite 0x03130b thru 0x03ffff (0xecf5 bytes) 000761 1734430740.646020 trunc from 0x040000 to 0x002540 (0x3dabf bytes) 000762 1734430740.659184 read 0x000304 thru 0x002540 (0x223d bytes) 000764 1734430740.665182 trunc from 0x002541 to 0x01a1d7 (0x17c97 bytes) 000765 1734430740.678327 mapread 0x00632d thru 0x00babf (0x5793 bytes) 000768 1734430740.686963 write 0x0293bb thru 0x029a6e (0x6b4 bytes) 000770 1734430740.688596 read 0x0011d7 thru 0x00ad06 (0x9b30 bytes) 000771 1734430740.693108 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000772 1734430740.830037 write 0x01beae thru 0x01f049 (0x319c bytes) 000773 1734430740.838623 mapread 0x006397 thru 0x015e53 (0xfabd bytes) 000775 1734430740.854945 MIRROR_EXTEND to 3 mirrors 000776 1734430741.236120 read 0x0140ce thru 0x01c121 (0x8054 bytes) 000777 1734430741.246980 write 0x01c09d thru 0x023c0c (0x7b70 bytes) 000778 1734430741.297580 mapread 0x005c40 thru 0x00c991 (0x6d52 bytes) 000779 1734430741.305274 mapwrite 0x027b64 thru 0x02aff5 (0x3492 bytes) 000783 1734430741.367847 mapread 0x01dddb thru 0x02332c (0x5552 bytes) 000787 1734430741.369369 read 0x01cd91 thru 0x020f09 (0x4179 bytes) 000788 1734430741.369755 mapread 0x00cb7b thru 0x00fcc4 (0x314a bytes) 000789 1734430741.370246 read 0x0148fb thru 0x0217da (0xcee0 bytes) 000790 1734430741.376052 mapread 0x01a95e thru 0x020fdd (0x6680 bytes) 000791 1734430741.377604 mapwrite 0x002071 thru 0x00dcfc (0xbc8c bytes) 000793 1734430741.420931 mapwrite 0x0363ac thru 0x03aa03 (0x4658 bytes) 000795 1734430741.491382 mapwrite 0x03c6a3 thru 0x03ffff (0x395d bytes) 000796 1734430741.555296 read 0x020062 thru 0x0202df (0x27e bytes) 000799 1734430741.557235 write 0x03bde3 thru 0x03ffff (0x421d bytes) 000801 1734430741.558558 write 0x02a014 thru 0x02dcca (0x3cb7 bytes) 000802 1734430741.567346 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000803 1734430741.680466 write 0x00e647 thru 0x014a71 (0x642b bytes) 000806 1734430741.689670 mapread 0x00fd6b thru 0x01fa11 (0xfca7 bytes) 000807 1734430741.698950 write 0x01f265 thru 0x023070 (0x3e0c bytes) 000808 1734430741.699706 mapread 0x0261c4 thru 0x02d88f (0x76cc bytes) 000809 1734430741.706241 trunc from 0x040000 to 0x00fead (0x30152 bytes) 000810 1734430741.720173 trunc from 0x00feae to 0x036677 (0x267ca bytes) 000812 1734430741.729508 read 0x021c15 thru 0x0287b0 (0x6b9c bytes) 000813 1734430741.735996 mapwrite 0x00b929 thru 0x013e63 (0x853b bytes) 000814 1734430741.778717 trunc from 0x036678 to 0x00b15a (0x2b51d bytes) 000816 1734430741.791190 write 0x03a357 thru 0x03ffff (0x5ca9 bytes) 000820 1734430741.792977 mapread 0x031c1b thru 0x037cec (0x60d2 bytes) 000821 1734430741.801201 read 0x019e53 thru 0x01d451 (0x35ff bytes) 000822 1734430741.808677 mapwrite 0x032794 thru 0x03ffff (0xd86c bytes) 000824 1734430741.858994 write 0x0272a0 thru 0x02dccf (0x6a30 bytes) 000826 1734430741.860085 trunc from 0x040000 to 0x02bcaf (0x14350 bytes) 000827 1734430741.873623 write 0x002d50 thru 0x00e77c (0xba2d bytes) 000828 1734430741.874819 mapread 0x002c17 thru 0x00d159 (0xa543 bytes) 000830 1734430741.877023 mapread 0x008ab7 thru 0x015e96 (0xd3e0 bytes) 000831 1734430741.879184 read 0x011446 thru 0x015a09 (0x45c4 bytes) 000833 1734430741.882882 MIRROR_RESYNC 2 mirrors 000836 1734430742.122583 read 0x025b9b thru 0x02bcaf (0x6115 bytes) 000841 1734430742.134740 MIRROR_EXTEND to 3 mirrors 000843 1734430742.432394 mapwrite 0x03064d thru 0x0364a7 (0x5e5b bytes) 000844 1734430742.525653 mapwrite 0x00c821 thru 0x01b72d (0xef0d bytes) 000846 1734430742.578009 write 0x0215b0 thru 0x02abee (0x963f bytes) 000847 1734430742.585715 mapwrite 0x00258c thru 0x012331 (0xfda6 bytes) 000850 1734430742.618293 read 0x02147a thru 0x0310d4 (0xfc5b bytes) 000851 1734430742.618965 mapwrite 0x005931 thru 0x013735 (0xde05 bytes) 000852 1734430742.653265 read 0x0053e7 thru 0x013f8d (0xeba7 bytes) 000853 1734430742.653598 mapread 0x00add1 thru 0x012675 (0x78a5 bytes) 000855 1734430742.654755 read 0x009967 thru 0x013855 (0x9eef bytes) 000858 1734430742.656255 read 0x014ddc thru 0x0170bc (0x22e1 bytes) 000859 1734430742.656662 trunc from 0x0364a8 to 0x02913c (0xd36b bytes) 000860 1734430742.670731 trunc from 0x02913d to 0x018765 (0x109d7 bytes) 000864 1734430742.683134 write 0x0374bf thru 0x037e4e (0x990 bytes) 000865 1734430742.683824 mapwrite 0x026a48 thru 0x029d5b (0x3314 bytes) 000866 1734430742.732421 write 0x02f3e4 thru 0x03d07b (0xdc98 bytes) 000867 1734430742.736512 MIRROR_RESYNC 3 mirrors 000869 1734430743.039064 trunc from 0x03d07c to 0x02cfb0 (0x100cb bytes) 000870 1734430743.105392 MIRROR_EXTEND to 4 mirrors 000871 1734430743.492719 MIRROR_RESYNC 4 mirrors 000872 1734430743.967739 mapread 0x00548c thru 0x010ba6 (0xb71b bytes) 000876 1734430743.986788 MIRROR_EXTEND to 5 mirrors 000877 1734430744.296098 mapwrite 0x027f84 thru 0x03344e (0xb4cb bytes) 000879 1734430744.428064 mapread 0x01de57 thru 0x027459 (0x9603 bytes) 000881 1734430744.437285 mapread 0x02a342 thru 0x02f0a1 (0x4d60 bytes) 000882 1734430744.438484 mapread 0x01ac6d thru 0x01fe7f (0x5213 bytes) 000885 1734430744.444363 MIRROR_EXTEND to 6 mirrors 000886 1734430744.760402 trunc from 0x03344f to 0x03bae6 (0x8698 bytes) 000888 1734430744.854152 MIRROR_RESYNC 6 mirrors 000889 1734430745.444155 write 0x015d2f thru 0x022363 (0xc635 bytes) 000890 1734430745.521737 mapread 0x027890 thru 0x02860f (0xd80 bytes) 000894 1734430745.528732 mapread 0x03ac8a thru 0x03bae6 (0xe5d bytes) 000895 1734430745.535415 read 0x00c674 thru 0x015285 (0x8c12 bytes) 000896 1734430745.540102 mapread 0x000c1d thru 0x00f138 (0xe51c bytes) 000897 1734430745.553307 trunc from 0x03bae7 to 0x02cd89 (0xed5d bytes) 000900 1734430745.566480 trunc from 0x02cd8a to 0x010216 (0x1cb73 bytes) 000903 1734430745.583548 write 0x02cf12 thru 0x03c20b (0xf2fa bytes) 000905 1734430745.585982 trunc from 0x03c20c to 0x01190a (0x2a901 bytes) 000906 1734430745.602771 read 0x00b925 thru 0x01190a (0x5fe6 bytes) 000909 1734430745.608591 write 0x03e383 thru 0x03ffff (0x1c7d bytes) 000910 1734430745.609546 read 0x036c37 thru 0x0376fa (0xac4 bytes) 000911 1734430745.616589 mapwrite 0x03890a thru 0x03ffff (0x76f6 bytes) 000912 1734430745.664592 write 0x0198cd thru 0x02676d (0xcea1 bytes) 000914 1734430745.666045 read 0x005980 thru 0x00edcb (0x944c bytes) 000915 1734430745.669536 MIRROR_RESYNC 6 mirrors 000919 1734430746.285588 mapwrite 0x028737 thru 0x02c9e7 (0x42b1 bytes) 000921 1734430746.448912 write 0x021ea9 thru 0x022df4 (0xf4c bytes) 000922 1734430746.456827 write 0x020a49 thru 0x028bbb (0x8173 bytes) 000923 1734430746.465793 MIRROR_EXTEND to 7 mirrors 000930 1734430746.777218 trunc from 0x040000 to 0x02343e (0x1cbc1 bytes) 000934 1734430746.856143 read 0x00a3ea thru 0x00cd5d (0x2974 bytes) 000938 1734430746.861563 write 0x01fbfa thru 0x029e28 (0xa22f bytes) 000939 1734430746.867640 read 0x020cd6 thru 0x029e28 (0x9153 bytes) 000942 1734430746.871641 trunc from 0x029e29 to 0x00d8a2 (0x1c586 bytes) 000943 1734430746.891037 mapread 0x00d66b thru 0x00d8a2 (0x238 bytes) 000946 1734430746.902394 trunc from 0x00d8a3 to 0x02afe3 (0x1d741 bytes) 000947 1734430746.918830 mapwrite 0x020939 thru 0x023f85 (0x364d bytes) 000948 1734430746.966735 mapwrite 0x035544 thru 0x03ffff (0xaabc bytes) 000949 1734430747.045251 mapwrite 0x0074fb thru 0x00ffd1 (0x8ad7 bytes) 000951 1734430747.108894 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000952 1734430747.234405 trunc from 0x040000 to 0x03df3c (0x20c3 bytes) 000953 1734430747.247283 write 0x01cfe0 thru 0x01d136 (0x157 bytes) 000955 1734430747.257040 mapread 0x035a09 thru 0x03df3c (0x8534 bytes) 000958 1734430747.270386 trunc from 0x03df3d to 0x002429 (0x3bb13 bytes) 000959 1734430747.280356 mapread 0x00240e thru 0x002429 (0x1c bytes) 000960 1734430747.285657 read 0x0001f4 thru 0x002429 (0x2236 bytes) 000961 1734430747.291214 mapread 0x002285 thru 0x002429 (0x1a5 bytes) 000962 1734430747.292543 write 0x02fe3d thru 0x037b78 (0x7d3c bytes) 000963 1734430747.293470 trunc from 0x037b79 to 0x007b57 (0x30021 bytes) 000967 1734430747.311955 write 0x00b8d6 thru 0x0169d8 (0xb103 bytes) 000969 1734430747.313514 mapread 0x00d163 thru 0x01177e (0x461c bytes) 000970 1734430747.314323 read 0x00f9a5 thru 0x0169d8 (0x7034 bytes) 000971 1734430747.316083 trunc from 0x0169d9 to 0x033431 (0x1ca59 bytes) 000972 1734430747.329530 write 0x001e04 thru 0x00ce4a (0xb047 bytes) 000974 1734430747.331171 mapread 0x02970b thru 0x02b806 (0x20fc bytes) 000975 1734430747.341806 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x403:0x0]: Invalid argument (22) 000976 1734430747.477239 read 0x01f0d4 thru 0x02b764 (0xc691 bytes) 000977 1734430747.482375 mapwrite 0x025ffc thru 0x034493 (0xe498 bytes) 000979 1734430747.548982 trunc from 0x034494 to 0x02f26b (0x5228 bytes) 000981 1734430747.558786 mapread 0x029fa3 thru 0x02f26b (0x52c9 bytes) 000986 1734430747.561271 mapwrite 0x002ebc thru 0x01043c (0xd581 bytes) 000987 1734430747.604697 mapwrite 0x01c572 thru 0x02732a (0xadb9 bytes) 000988 1734430747.644586 MIRROR_EXTEND to 6 mirrors 000989 1734430747.924340 read 0x01864f thru 0x02638b (0xdd3d bytes) 000991 1734430747.932700 trunc from 0x02f26c to 0x02ba17 (0x3854 bytes) 000993 1734430747.990411 mapwrite 0x0223a5 thru 0x02f27c (0xced8 bytes) 000994 1734430748.067443 read 0x002c44 thru 0x00695b (0x3d18 bytes) 000997 1734430748.074082 mapread 0x01ff30 thru 0x02a7a3 (0xa874 bytes) 000999 1734430748.075480 mapread 0x01a153 thru 0x023f1c (0x9dca bytes) 001000 1734430748.076761 mapread 0x0168c6 thru 0x022d73 (0xc4ae bytes) All operations completed A-OK! PASS 70a (34s) == sanity-flr test 200a: stress test ===================== 05:19:10 (1734430750) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6681120 .. Extending file size to 4434624 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8529024 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7301952 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7376128 .. Extending file size to 7719296 .. Extending file size to 7752448 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8483456 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8857056 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9195968 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8501536 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8845824 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 27464 27465 27466 27467 27469 failed 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) 192.168.202.103@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:20:16 (1734430816) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5505472 .. Extending file size to 2284672 .. failed Extending file size to 7585120 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8208864 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5900192 .. Extending file size to 8588736 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6442016 .. Extending file size to 8772064 .. Extending file size to 8819168 .. Extending file size to 6618304 .. Extending file size to 6922144 .. Extending file size to 7942080 .. Extending file size to 8574752 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8824352 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8957792 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9081440 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 29497 29498 29500 29501 29503 done 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) 192.168.202.103@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:21:20 (1734430880) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.038238 s, 54.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.102071 s, 20.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:21:26 (1734430886) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0277217 s, 75.7 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 05:21:27 (1734430887) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x440:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ac:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ac:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ad:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x4af:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:21:29 (1734430889) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.033211 s, 63.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:21:32 (1734430892) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0418043 s, 50.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b3:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:21:34 (1734430894) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0283796 s, 73.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b5:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:21:57 (1734430917) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg203-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0356894 s, 29.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b3:0x0] } PASS 204d (47s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:22:44 (1734430964) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg203-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0537583 s, 39.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:23:31 (1734431011) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg203-server mds-ost sync done. Creating new pool test_204f oleg203-server: Pool lustre.test_204f created Adding targets to pool oleg203-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg203-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0528121 s, 39.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4b9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg203-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg203-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg203-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (58s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:24:29 (1734431069) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:24:32 (1734431072) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:24:35 (1734431075) oleg203-server: Pool lustre.test_206 created Adding targets to pool oleg203-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg203-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg203-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg203-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg203-server: Pool lustre.test_206 destroyed PASS 206 (17s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:24:52 (1734431092) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0296837 s, 35.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133149 s, 78.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x79187cf4 Mirror 3: 0x79187cf4 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:24:56 (1734431096) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:24:57 (1734431097) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:24:59 (1734431099) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0587144 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1100000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x6d0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x6d0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x6d0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1430000 Read this (4 bytes): BBBB 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) PASS 209a (20s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:25:19 (1734431119) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00734878 s, 1.1 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:25:21 (1734431121) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0194739 s, 53.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x465 lmm_fid: [0x200000404:0x465:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x465 lmm_fid: [0x200000404:0x465:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c3:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835045 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-17 05:25:22.000000000 -0500 Modify: 2024-12-17 05:25:22.000000000 -0500 Change: 2024-12-17 05:25:23.000000000 -0500 Birth: - PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:25:24 (1734431124) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:25:26 (1734431126) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.286451 s, 143 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0262109 s, 156 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 17 05:25 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x467:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 17 05:25 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 901 sec ============ 05:25:29 (1734431129) === sanity-flr: start cleanup 05:25:30 (1734431130) === === sanity-flr: finish cleanup 05:25:33 (1734431133) ===