-----============= acceptance-small: sanity-flr ============----- Thu Dec 5 06:13:26 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c 49a === sanity-flr: start setup 06:13:30 (1733397210) === oleg233-client.virtnet: executing check_config_client /mnt/lustre oleg233-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg233-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012da6c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012da6c800.idle_timeout=debug disable quota as required oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 06:13:37 (1733397217) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6952] == sanity-flr test 0a: lfs mirror create with -N option == 06:13:38 (1733397218) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:13:41 (1733397221) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:13:42 (1733397222) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:13:42 (1733397222) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:13:46 (1733397226) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:13:47 (1733397227) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 06:13:48 (1733397228) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012da6c800.stats=clear osc.lustre-OST0001-osc-ffff88012da6c800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:13:50 (1733397230) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:13:52 (1733397232) PASS 0j (1s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:13:53 (1733397233) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:13:54 (1733397234) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 06:13:56 (1733397236) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:13:57 (1733397237) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:13:58 (1733397238) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:13:59 (1733397239) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:14:01 (1733397241) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:14:03 (1733397243) 13+0 records in 13+0 records out 13631488 bytes (14 MB) copied, 0.361199 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375908 s, 27.9 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:14:05 (1733397245) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0296688 s, 177 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:14:07 (1733397247) Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012da6c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012da6c800.ost_server_uuid in DISCONN state after 3 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012da6c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012da6c800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (21s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:14:28 (1733397268) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0813846 s, 77.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012da6c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012da6c800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:14:37 (1733397277) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012da6c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012da6c800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012da6c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012da6c800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (16s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:14:53 (1733397293) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.704353 s, 74.4 MB/s -rw-r--r-- 1 root root 52428800 Dec 5 06:14 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.176s user 0m0.000s sys 0m0.155s Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012da6c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012da6c800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.181s user 0m0.000s sys 0m0.111s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012da6c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012da6c800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.166s user 0m0.001s sys 0m0.105s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012da6c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012da6c800.ost_server_uuid in FULL state after 0 sec PASS 33b (16s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:15:10 (1733397310) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:15:10 (1733397310) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:15:11 (1733397311) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 06:15:12 (1733397312) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 06:15:13 (1733397313) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.111643 s, 113 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:15:16 (1733397316) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00798933 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01125 s, 93.2 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:15:17 (1733397317) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:15:20 (1733397320) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000621649 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000564669 s, 1.8 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000703795 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000560752 s, 1.8 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 06:15:22 (1733397322) create test files with size 15744500 .. 75d90bf8731c62f07c7b9f2c25e8c67c - 83fd2c0fb588dd0b46d03242ff69c7df - f78f918eda114d5d91b9785193a796ba - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 06:15:27 (1733397327) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:15:29 (1733397329) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:15:30 (1733397330) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200686 s, 52.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00817937 s, 128 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 06:15:32 (1733397332) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0245999 s, 85.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0747673 s, 56.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0382755 s, 54.8 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 06:15:34 (1733397334) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 06:15:35 (1733397335) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:15:36 (1733397336) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00910096 s, 115 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) == sanity-flr test 44a: lfs mirror split check =========== 06:15:37 (1733397337) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:15:38 (1733397338) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4d:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x13:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:15:40 (1733397340) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0557281 s, 188 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:15:42 (1733397342) llite.lustre-ffff8800a268c000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0547616 s, 191 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff880130c36800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:15:45 (1733397345) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:15:52 (1733397352) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:15:58 (1733397358) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 06:15:59 (1733397359) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0531126 s, 59.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0420203 s, 74.9 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 3801edd62d6d53c5f65f116d174ed146 - mirror 1 checksum: 2988246f3e591ae412302a6e1209ef53 - mirror 2 checksum: 3801edd62d6d53c5f65f116d174ed146 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Dec 5 06:16:00 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Dec 5 06:16:00 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3801edd62d6d53c5f65f116d174ed146 - mirror 1 checksum: 2988246f3e591ae412302a6e1209ef53 - mirror 2 checksum: 2988246f3e591ae412302a6e1209ef53 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Dec 5 06:16:00 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Dec 5 06:16:00 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:16:01 (1733397361) Failing mds1 on oleg233-server Stopping /mnt/lustre-mds1 (opts:) on oleg233-server 06:16:03 (1733397363) shut down Failover mds1 to oleg233-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0000 06:16:15 (1733397375) targets are mounted 06:16:15 (1733397375) facet_failover done oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00806579 s, 130 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (19s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:16:20 (1733397380) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (0s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:16:20 (1733397380) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:16:21 (1733397381) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00192602 s, 8.5 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:16:23 (1733397383) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (0s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:16:23 (1733397383) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:16:24 (1733397384) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:16:28 (1733397388) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (17s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:16:45 (1733397405) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:17:02 (1733397422) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:17:19 (1733397439) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 06:17:20 (1733397440) mirror create pid 22174 mirror split pid 22175 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4855: 22174 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4855: 22175 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 06:18:22 (1733397502) Seed set to 7502 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x6cf5 000002 1733397502.597120 trunc from 00000000 to 0x006cf4 (0x6cf5 bytes) truncating to largest ever: 0x3d97b 000007 1733397502.601521 trunc from 0x006cf5 to 0x03d97a (0x36c86 bytes) 000008 1733397502.605208 mapread 0x00da15 thru 0x01bce9 (0xe2d5 bytes) 000010 1733397502.611955 trunc from 0x03d97b to 0x00bfde (0x3199c bytes) 000011 1733397502.615406 trunc from 0x00bfdf to 0x00a3f3 (0x1beb bytes) 000013 1733397502.618721 mapread 0x003514 thru 0x006ac7 (0x35b4 bytes) 000015 1733397502.621966 trunc from 0x00a3f4 to 0x00c0e0 (0x1ced bytes) 000017 1733397502.625809 write 0x027c7a thru 0x028a28 (0xdaf bytes) 000021 1733397502.627223 mapread 0x018c23 thru 0x0198af (0xc8d bytes) 000024 1733397502.631383 mapread 0x0063bf thru 0x00da34 (0x7676 bytes) 000025 1733397502.633890 trunc from 0x028a29 to 0x0237ed (0x523b bytes) 000027 1733397502.638405 write 0x02768e thru 0x02c2b5 (0x4c28 bytes) 000028 1733397502.639178 write 0x00a7e5 thru 0x00da3c (0x3258 bytes) 000029 1733397502.642009 read 0x0085e8 thru 0x00db45 (0x555e bytes) 000030 1733397502.642379 write 0x0395fa thru 0x03ffff (0x6a06 bytes) 000031 1733397502.643074 trunc from 0x040000 to 0x012f09 (0x2d0f6 bytes) 000033 1733397502.647810 write 0x0236b2 thru 0x0278a6 (0x41f5 bytes) 000039 1733397502.649965 mapread 0x01d296 thru 0x01d54f (0x2ba bytes) 000041 1733397502.652991 read 0x01e8ad thru 0x020660 (0x1db4 bytes) 000042 1733397502.653377 read 0x022e5e thru 0x025474 (0x2617 bytes) 000044 1733397502.654004 mapread 0x0169ee thru 0x017c27 (0x123a bytes) 000045 1733397502.654461 trunc from 0x0278a7 to 0x01e890 (0x9016 bytes) 000047 1733397502.658251 trunc from 0x01e891 to 0x0047ad (0x1a0e3 bytes) 000049 1733397502.661816 write 0x03801c thru 0x03ffff (0x7fe4 bytes) 000051 1733397502.664224 MIRROR_EXTEND to 2 mirrors 000052 1733397502.770265 mapread 0x027c0b thru 0x02e45b (0x6851 bytes) 000054 1733397502.775445 write 0x008407 thru 0x00e73b (0x6335 bytes) 000056 1733397502.791577 trunc from 0x040000 to 0x0137ce (0x2c831 bytes) 000057 1733397502.799015 mapread 0x012462 thru 0x0137ce (0x136d bytes) 000058 1733397502.800677 write 0x0324c3 thru 0x03ffff (0xdb3d bytes) 000059 1733397502.801448 trunc from 0x040000 to 0x01eec0 (0x2113f bytes) 000060 1733397502.804628 mapread 0x00ff34 thru 0x010807 (0x8d4 bytes) 000061 1733397502.805050 read 0x01db06 thru 0x01eec0 (0x13bb bytes) 000062 1733397502.806564 mapread 0x00d3e5 thru 0x0191b7 (0xbdd3 bytes) 000063 1733397502.808759 mapwrite 0x00e2d0 thru 0x00f073 (0xda4 bytes) 000064 1733397502.820027 trunc from 0x01eec1 to 0x00bcda (0x131e6 bytes) 000065 1733397502.823821 mapread 0x00b5ae thru 0x00bcda (0x72d bytes) 000067 1733397502.825672 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000070 1733397502.856926 MIRROR_EXTEND to 2 mirrors 000071 1733397502.951238 write 0x0053cd thru 0x00894b (0x357f bytes) 000073 1733397502.966153 write 0x03ccf9 thru 0x03ffff (0x3307 bytes) 000075 1733397502.967022 mapwrite 0x000b9e thru 0x003c85 (0x30e8 bytes) 000076 1733397502.986387 mapread 0x013896 thru 0x01b93e (0x80a9 bytes) 000077 1733397502.991727 MIRROR_EXTEND to 3 mirrors 000078 1733397503.101646 read 0x03858e thru 0x03d183 (0x4bf6 bytes) 000079 1733397503.105282 trunc from 0x040000 to 0x0170db (0x28f24 bytes) 000083 1733397503.123085 MIRROR_EXTEND to 4 mirrors 000085 1733397503.229000 MIRROR_RESYNC 4 mirrors 000086 1733397503.345292 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000088 1733397503.402183 write 0x0248c0 thru 0x024b20 (0x261 bytes) 000089 1733397503.416908 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000090 1733397503.462528 write 0x011b31 thru 0x014662 (0x2b32 bytes) 000092 1733397503.465403 read 0x01874c thru 0x018e28 (0x6dd bytes) 000094 1733397503.468224 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000097 1733397503.497425 mapwrite 0x01c4dc thru 0x027d8d (0xb8b2 bytes) 000101 1733397503.522622 mapread 0x0191ee thru 0x01f151 (0x5f64 bytes) 000103 1733397503.523550 mapread 0x012967 thru 0x0218cb (0xef65 bytes) 000105 1733397503.526370 trunc from 0x027d8e to 0x0158d9 (0x124b4 bytes) 000106 1733397503.529729 read 0x0030be thru 0x006bbe (0x3b01 bytes) 000109 1733397503.532071 write 0x03e8f9 thru 0x03ffff (0x1707 bytes) 000110 1733397503.532722 trunc from 0x040000 to 0x00023c (0x3fdc3 bytes) 000111 1733397503.536067 mapread 0x0001fc thru 0x00023c (0x41 bytes) 000114 1733397503.538002 mapread 0x0001f9 thru 0x00023c (0x44 bytes) 000117 1733397503.538825 mapread 0x00002d thru 0x00023c (0x210 bytes) 000120 1733397503.539607 write 0x013396 thru 0x02198d (0xe5f8 bytes) 000121 1733397503.540496 mapwrite 0x028550 thru 0x028f63 (0xa14 bytes) 000122 1733397503.558364 read 0x00d879 thru 0x00de52 (0x5da bytes) 000123 1733397503.561072 read 0x026d3a thru 0x028f63 (0x222a bytes) 000128 1733397503.562420 write 0x02f772 thru 0x031e83 (0x2712 bytes) 000129 1733397503.562938 mapwrite 0x03890c thru 0x03ffff (0x76f4 bytes) 000130 1733397503.583146 mapwrite 0x00a153 thru 0x012493 (0x8341 bytes) 000131 1733397503.595453 trunc from 0x040000 to 0x0211a3 (0x1ee5c bytes) 000132 1733397503.599949 mapwrite 0x0359ae thru 0x03ffff (0xa652 bytes) 000135 1733397503.631745 trunc from 0x040000 to 0x01bbbb (0x24444 bytes) 000137 1733397503.635968 read 0x009635 thru 0x017ab1 (0xe47d bytes) 000140 1733397503.637056 trunc from 0x01bbbc to 0x0178d3 (0x42e8 bytes) 000141 1733397503.640394 mapread 0x0027a2 thru 0x005bd7 (0x3436 bytes) 000143 1733397503.641171 write 0x037afb thru 0x03b9ba (0x3ec0 bytes) 000144 1733397503.641809 mapread 0x020077 thru 0x0262cc (0x6256 bytes) 000146 1733397503.645261 trunc from 0x03b9bb to 0x015e6b (0x25b4f bytes) 000147 1733397503.649396 read 0x009d03 thru 0x00f3e1 (0x56df bytes) 000149 1733397503.650238 trunc from 0x015e6c to 0x0223fd (0xc592 bytes) 000150 1733397503.654014 mapread 0x00e1a0 thru 0x016ad9 (0x893a bytes) 000154 1733397503.657531 mapread 0x01477f thru 0x0223fd (0xdc7f bytes) 000155 1733397503.658040 mapread 0x00fe1c thru 0x01aff0 (0xb1d5 bytes) 000158 1733397503.659044 mapwrite 0x01ce51 thru 0x02c9a1 (0xfb51 bytes) 000159 1733397503.680891 trunc from 0x02c9a2 to 0x00def7 (0x1eaaa bytes) 000160 1733397503.685037 mapwrite 0x00d0b5 thru 0x01cac1 (0xfa0d bytes) 000161 1733397503.725821 read 0x017a3c thru 0x019d0f (0x22d4 bytes) 000162 1733397503.726170 mapwrite 0x02f2ca thru 0x038810 (0x9547 bytes) 000163 1733397503.747783 mapwrite 0x019c37 thru 0x025180 (0xb54a bytes) 000164 1733397503.761433 write 0x00071a thru 0x008db5 (0x869c bytes) 000166 1733397503.762809 write 0x001f2a thru 0x00f5b6 (0xd68d bytes) 000168 1733397503.763833 write 0x02e2b5 thru 0x0368fb (0x8647 bytes) 000169 1733397503.764346 write 0x0361e1 thru 0x038f5a (0x2d7a bytes) 000170 1733397503.764809 mapread 0x0167cc thru 0x01fafc (0x9331 bytes) 000172 1733397503.765431 write 0x02c9be thru 0x037df4 (0xb437 bytes) 000175 1733397503.766390 write 0x00ab47 thru 0x00e12f (0x35e9 bytes) 000177 1733397503.767000 read 0x00f22c thru 0x0148a2 (0x5677 bytes) 000178 1733397503.767264 mapwrite 0x03cc4f thru 0x03ffff (0x33b1 bytes) 000179 1733397503.789218 trunc from 0x040000 to 0x0293f1 (0x16c0e bytes) 000180 1733397503.793376 write 0x0163b5 thru 0x01f016 (0x8c62 bytes) 000181 1733397503.794021 trunc from 0x0293f2 to 0x029939 (0x548 bytes) 000182 1733397503.797547 write 0x016578 thru 0x018989 (0x2412 bytes) 000183 1733397503.798058 trunc from 0x02993a to 0x0205f5 (0x9344 bytes) 000184 1733397503.801984 mapread 0x014ce6 thru 0x0205f5 (0xb910 bytes) 000185 1733397503.803635 MIRROR_EXTEND to 2 mirrors 000188 1733397503.900908 MIRROR_RESYNC 2 mirrors 000189 1733397503.933254 mapwrite 0x002331 thru 0x00ab8e (0x885e bytes) 000193 1733397503.979398 MIRROR_EXTEND to 3 mirrors 000194 1733397504.085233 mapread 0x001d8b thru 0x00db1d (0xbd93 bytes) 000197 1733397504.098729 MIRROR_RESYNC 3 mirrors 000198 1733397504.184459 trunc from 0x0205f6 to 0x012dfe (0xd7f7 bytes) 000206 1733397504.202952 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000209 1733397504.249594 mapread 0x00b05f thru 0x012dfe (0x7da0 bytes) 000215 1733397504.255717 trunc from 0x012dff to 0x024e90 (0x12092 bytes) 000218 1733397504.260259 write 0x012a5e thru 0x015b64 (0x3107 bytes) 000219 1733397504.261998 read 0x007466 thru 0x00e183 (0x6d1e bytes) 000220 1733397504.263618 mapread 0x00b4a0 thru 0x015f7e (0xaadf bytes) 000221 1733397504.264332 write 0x021af1 thru 0x022e4d (0x135d bytes) 000222 1733397504.266816 read 0x01fa4f thru 0x024e90 (0x5442 bytes) 000223 1733397504.267186 mapwrite 0x014e64 thru 0x023cb4 (0xee51 bytes) 000225 1733397504.283986 write 0x004cb2 thru 0x011102 (0xc451 bytes) 000227 1733397504.285451 mapread 0x0061d8 thru 0x014b98 (0xe9c1 bytes) 000228 1733397504.286152 mapread 0x0011f5 thru 0x011131 (0xff3d bytes) 000230 1733397504.287221 mapread 0x01d4d7 thru 0x024e90 (0x79ba bytes) 000238 1733397504.290582 mapwrite 0x03af24 thru 0x03ffff (0x50dc bytes) 000239 1733397504.309531 mapwrite 0x03c66d thru 0x03ffff (0x3993 bytes) 000240 1733397504.319593 write 0x022664 thru 0x02a020 (0x79bd bytes) 000241 1733397504.320357 mapread 0x023301 thru 0x02acae (0x79ae bytes) 000242 1733397504.320953 mapread 0x0141d2 thru 0x016ceb (0x2b1a bytes) 000243 1733397504.321390 trunc from 0x040000 to 0x017af8 (0x28507 bytes) 000245 1733397504.326954 MIRROR_EXTEND to 3 mirrors 000248 1733397504.413423 mapread 0x0146de thru 0x017af8 (0x341b bytes) 000249 1733397504.419317 write 0x02c7f1 thru 0x03a122 (0xd932 bytes) 000250 1733397504.435369 MIRROR_EXTEND to 4 mirrors 000251 1733397504.538913 mapread 0x0274fe thru 0x02d014 (0x5b17 bytes) 000252 1733397504.545472 read 0x009f8b thru 0x017887 (0xd8fd bytes) 000253 1733397504.548723 read 0x02e329 thru 0x03a122 (0xbdfa bytes) 000255 1733397504.553342 mapwrite 0x02893c thru 0x02a849 (0x1f0e bytes) 000257 1733397504.593087 trunc from 0x03a123 to 0x030d0c (0x9416 bytes) 000259 1733397504.601843 trunc from 0x030d0d to 0x0287ac (0x8560 bytes) 000262 1733397504.605998 mapread 0x016d59 thru 0x022552 (0xb7fa bytes) 000263 1733397504.620595 write 0x03013e thru 0x03a093 (0x9f56 bytes) 000265 1733397504.623078 MIRROR_EXTEND to 5 mirrors 000266 1733397504.720810 mapread 0x002e12 thru 0x003c1a (0xe09 bytes) 000267 1733397504.725256 mapwrite 0x02183a thru 0x025a64 (0x422b bytes) 000268 1733397504.772379 mapread 0x03167a thru 0x0356b2 (0x4039 bytes) 000269 1733397504.776516 MIRROR_RESYNC 5 mirrors 000271 1733397504.928147 mapwrite 0x02bfb7 thru 0x02c3bd (0x407 bytes) 000272 1733397504.970516 trunc from 0x03a094 to 0x021585 (0x18b0e bytes) 000273 1733397504.979034 mapwrite 0x014f6b thru 0x022490 (0xd526 bytes) 000276 1733397505.001982 read 0x01aeb8 thru 0x022490 (0x75d9 bytes) 000277 1733397505.002423 mapread 0x01c9b5 thru 0x022490 (0x5adc bytes) 000278 1733397505.002916 write 0x0249ba thru 0x0283ff (0x3a46 bytes) 000280 1733397505.005063 MIRROR_RESYNC 5 mirrors 000281 1733397505.147897 mapwrite 0x03e0d1 thru 0x03ffff (0x1f2f bytes) 000285 1733397505.180815 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000287 1733397505.225130 read 0x014fc4 thru 0x01bf1d (0x6f5a bytes) 000292 1733397505.228640 write 0x02accc thru 0x033205 (0x853a bytes) 000293 1733397505.231569 write 0x013276 thru 0x021df8 (0xeb83 bytes) 000294 1733397505.236202 MIRROR_EXTEND to 5 mirrors 000295 1733397505.344367 trunc from 0x040000 to 0x027edc (0x18123 bytes) 000296 1733397505.363621 mapwrite 0x03c2ed thru 0x03ffff (0x3d13 bytes) 000297 1733397505.383823 trunc from 0x040000 to 0x03a91e (0x56e1 bytes) 000298 1733397505.387266 mapwrite 0x01c1f4 thru 0x024118 (0x7f25 bytes) 000300 1733397505.400851 write 0x03c186 thru 0x03edea (0x2c65 bytes) 000301 1733397505.401395 read 0x004ef1 thru 0x00aa45 (0x5b55 bytes) 000302 1733397505.403493 mapread 0x02e2c9 thru 0x03959a (0xb2d2 bytes) 000304 1733397505.409389 MIRROR_EXTEND to 6 mirrors 000305 1733397505.500215 trunc from 0x03edeb to 0x00c570 (0x3287a bytes) 000306 1733397505.517839 MIRROR_EXTEND to 7 mirrors 000307 1733397505.603871 trunc from 0x00c571 to 0x02fc66 (0x236f6 bytes) 000309 1733397505.621460 write 0x0264ac thru 0x02effd (0x8b52 bytes) 000311 1733397505.625433 MIRROR_RESYNC 7 mirrors 000312 1733397505.826164 MIRROR_RESYNC 7 mirrors 000314 1733397505.876004 mapread 0x000e8b thru 0x003103 (0x2279 bytes) 000315 1733397505.882820 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000316 1733397505.948841 mapread 0x005401 thru 0x00d744 (0x8344 bytes) 000319 1733397505.955201 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000320 1733397506.026929 mapread 0x015d2f thru 0x01baa3 (0x5d75 bytes) 000322 1733397506.032168 mapread 0x00e0d5 thru 0x0172b5 (0x91e1 bytes) 000323 1733397506.034546 read 0x01373d thru 0x01fe06 (0xc6ca bytes) 000325 1733397506.035045 mapwrite 0x01a5c6 thru 0x02476e (0xa1a9 bytes) 000326 1733397506.080955 write 0x03b9d5 thru 0x03ffff (0x462b bytes) 000328 1733397506.082107 mapwrite 0x018fed thru 0x01d5b9 (0x45cd bytes) 000331 1733397506.099179 MIRROR_EXTEND to 6 mirrors 000332 1733397506.202917 write 0x0398ad thru 0x03ffff (0x6753 bytes) 000334 1733397506.221885 MIRROR_EXTEND to 7 mirrors 000335 1733397506.336817 read 0x03a3f8 thru 0x03d189 (0x2d92 bytes) 000336 1733397506.339331 read 0x007fef thru 0x0111ba (0x91cc bytes) 000338 1733397506.344316 trunc from 0x040000 to 0x020582 (0x1fa7d bytes) 000339 1733397506.366027 MIRROR_EXTEND to 8 mirrors 000340 1733397506.461905 trunc from 0x020583 to 0x00413c (0x1c446 bytes) 000343 1733397506.482786 write 0x02cd0c thru 0x031571 (0x4866 bytes) 000346 1733397506.483856 mapwrite 0x03a46b thru 0x03ffff (0x5b95 bytes) 000348 1733397506.509772 write 0x005dd0 thru 0x012b64 (0xcd95 bytes) 000349 1733397506.514081 mapread 0x00dc68 thru 0x0100bf (0x2458 bytes) 000352 1733397506.515015 mapread 0x01a734 thru 0x01fa41 (0x530e bytes) 000356 1733397506.519883 write 0x004276 thru 0x004ea2 (0xc2d bytes) 000357 1733397506.521495 MIRROR_EXTEND to 9 mirrors 000358 1733397506.636822 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000363 1733397506.719406 read 0x00aac7 thru 0x010282 (0x57bc bytes) 000364 1733397506.724318 mapwrite 0x013067 thru 0x01fbec (0xcb86 bytes) 000365 1733397506.776614 mapwrite 0x0013dd thru 0x00edcc (0xd9f0 bytes) 000366 1733397506.792969 mapwrite 0x000d4a thru 0x00ec1c (0xded3 bytes) 000369 1733397506.809296 mapwrite 0x035675 thru 0x03f253 (0x9bdf bytes) 000370 1733397506.827211 read 0x025493 thru 0x02bf24 (0x6a92 bytes) 000371 1733397506.830295 mapread 0x004068 thru 0x0053af (0x1348 bytes) 000373 1733397506.831337 trunc from 0x040000 to 0x033345 (0xccba bytes) 000374 1733397506.840639 mapwrite 0x02c06d thru 0x036c02 (0xab96 bytes) 000377 1733397506.862708 trunc from 0x036c03 to 0x02b3e6 (0xb81c bytes) 000379 1733397506.866432 mapwrite 0x0189aa thru 0x01b5c4 (0x2c1b bytes) 000380 1733397506.879588 mapread 0x026b5b thru 0x02777d (0xc23 bytes) 000381 1733397506.880114 mapread 0x00cd14 thru 0x01c576 (0xf863 bytes) 000382 1733397506.880742 trunc from 0x02b3e7 to 0x0286dd (0x2d09 bytes) 000383 1733397506.884971 trunc from 0x0286de to 0x006a4d (0x21c90 bytes) 000384 1733397506.888887 mapwrite 0x01b59a thru 0x021dfb (0x6862 bytes) 000385 1733397506.916905 MIRROR_EXTEND to 9 mirrors 000386 1733397507.015836 read 0x01f92f thru 0x021dfb (0x24cd bytes) 000387 1733397507.018129 read 0x008210 thru 0x015e13 (0xdc04 bytes) 000389 1733397507.023504 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000390 1733397507.087331 mapwrite 0x03357d thru 0x03def4 (0xa978 bytes) 000391 1733397507.128338 trunc from 0x03def5 to 0x02e242 (0xfcb2 bytes) 000394 1733397507.133826 write 0x03e0f4 thru 0x03ffff (0x1f0c bytes) 000396 1733397507.135851 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000397 1733397507.191181 MIRROR_EXTEND to 8 mirrors 000398 1733397507.303666 trunc from 0x040000 to 0x01a06f (0x25f90 bytes) 000399 1733397507.320285 MIRROR_RESYNC 8 mirrors 000400 1733397507.559116 write 0x01b899 thru 0x02b1b4 (0xf91c bytes) 000401 1733397507.573355 write 0x03ab30 thru 0x03b155 (0x626 bytes) 000402 1733397507.574057 trunc from 0x03b156 to 0x00e17e (0x2cfd7 bytes) 000403 1733397507.578789 mapread 0x000348 thru 0x004523 (0x41dc bytes) 000404 1733397507.584831 trunc from 0x00e17f to 0x02f205 (0x21087 bytes) 000405 1733397507.588697 read 0x01731e thru 0x022cce (0xb9b1 bytes) 000408 1733397507.591454 read 0x007233 thru 0x007251 (0x1f bytes) 000410 1733397507.591946 write 0x0216a5 thru 0x02a70b (0x9067 bytes) 000411 1733397507.594506 mapwrite 0x0043d1 thru 0x012802 (0xe432 bytes) 000413 1733397507.610112 trunc from 0x02f206 to 0x02a5ba (0x4c4b bytes) 000414 1733397507.613868 mapwrite 0x00c438 thru 0x01ad52 (0xe91b bytes) 000415 1733397507.630816 mapwrite 0x01f825 thru 0x02d088 (0xd864 bytes) 000416 1733397507.651928 read 0x0047f8 thru 0x00a6d0 (0x5ed9 bytes) 000417 1733397507.652360 mapwrite 0x00fb06 thru 0x014e6f (0x536a bytes) 000418 1733397507.664468 write 0x020371 thru 0x023958 (0x35e8 bytes) 000424 1733397507.666253 trunc from 0x02d089 to 0x001070 (0x2c018 bytes) 000425 1733397507.670934 mapread 0x000f71 thru 0x001070 (0x100 bytes) 000427 1733397507.672034 read 0x0006cf thru 0x001070 (0x9a2 bytes) 000429 1733397507.672907 mapwrite 0x010784 thru 0x013926 (0x31a3 bytes) 000430 1733397507.693362 MIRROR_EXTEND to 9 mirrors 000432 1733397507.788827 MIRROR_RESYNC 9 mirrors 000435 1733397508.002834 mapwrite 0x025cba thru 0x02a560 (0x48a7 bytes) 000437 1733397508.044790 mapread 0x01b8aa thru 0x0226d9 (0x6e30 bytes) 000438 1733397508.047779 read 0x017750 thru 0x01d40a (0x5cbb bytes) 000439 1733397508.050887 MIRROR_RESYNC 9 mirrors 000440 1733397508.277071 mapread 0x006f33 thru 0x00e242 (0x7310 bytes) 000441 1733397508.281609 trunc from 0x02a561 to 0x0393a5 (0xee45 bytes) 000442 1733397508.302437 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000444 1733397508.352720 trunc from 0x0393a6 to 0x033d63 (0x5642 bytes) 000448 1733397508.358430 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000452 1733397508.406005 mapread 0x00da50 thru 0x01847c (0xaa2d bytes) 000453 1733397508.412522 read 0x00a652 thru 0x00e92c (0x42db bytes) 000454 1733397508.412837 write 0x017157 thru 0x01af01 (0x3dab bytes) 000455 1733397508.413315 read 0x016bc9 thru 0x01a09f (0x34d7 bytes) 000456 1733397508.413570 mapwrite 0x0275d6 thru 0x02c489 (0x4eb4 bytes) 000457 1733397508.430167 mapwrite 0x0233ac thru 0x02b0ee (0x7d43 bytes) 000460 1733397508.443621 mapwrite 0x039c89 thru 0x03ffff (0x6377 bytes) 000462 1733397508.462088 trunc from 0x040000 to 0x00543c (0x3abc3 bytes) 000464 1733397508.466612 mapread 0x001eda thru 0x00543c (0x3563 bytes) 000465 1733397508.467108 read 0x000b02 thru 0x00543c (0x493b bytes) 000468 1733397508.467979 mapread 0x003f18 thru 0x00543c (0x1525 bytes) 000470 1733397508.468783 write 0x018a51 thru 0x0268ba (0xde6a bytes) 000472 1733397508.469944 write 0x00351e thru 0x010fe3 (0xdac6 bytes) 000473 1733397508.471838 read 0x00d6fa thru 0x0184c2 (0xadc9 bytes) 000475 1733397508.474246 mapread 0x0055a9 thru 0x00d664 (0x80bc bytes) 000478 1733397508.475285 trunc from 0x0268bb to 0x032224 (0xb96a bytes) 000480 1733397508.479274 write 0x02d328 thru 0x033922 (0x65fb bytes) 000481 1733397508.481714 read 0x012548 thru 0x013d7a (0x1833 bytes) 000484 1733397508.482411 mapread 0x01db92 thru 0x025d0a (0x8179 bytes) 000485 1733397508.482824 mapwrite 0x005c7f thru 0x015aa5 (0xfe27 bytes) 000488 1733397508.500314 trunc from 0x033923 to 0x01a9ee (0x18f34 bytes) 000489 1733397508.505062 mapread 0x019f6e thru 0x01a9ee (0xa81 bytes) 000495 1733397508.506890 write 0x0271e9 thru 0x02f66f (0x8487 bytes) 000497 1733397508.508126 mapread 0x024aeb thru 0x02b3d6 (0x68ec bytes) 000498 1733397508.510926 write 0x03dc45 thru 0x03fb7d (0x1f39 bytes) 000502 1733397508.512109 trunc from 0x03fb7e to 0x029c8b (0x15ef2 bytes) 000503 1733397508.516744 trunc from 0x029c8c to 0x009024 (0x20c67 bytes) 000504 1733397508.520801 mapread 0x008495 thru 0x008d1d (0x889 bytes) 000505 1733397508.521435 mapwrite 0x023227 thru 0x024b79 (0x1953 bytes) 000507 1733397508.539520 mapread 0x00bcbc thru 0x00efdb (0x3320 bytes) 000508 1733397508.544754 trunc from 0x024b7a to 0x018677 (0xc502 bytes) 000509 1733397508.548701 read 0x001267 thru 0x00bdcd (0xab67 bytes) 000510 1733397508.551047 read 0x006886 thru 0x0071cf (0x94a bytes) 000513 1733397508.551778 trunc from 0x018678 to 0x014815 (0x3e62 bytes) 000514 1733397508.556488 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000516 1733397508.603716 mapread 0x007161 thru 0x01163a (0xa4da bytes) 000517 1733397508.610186 mapread 0x00d577 thru 0x014815 (0x729f bytes) 000519 1733397508.610830 mapwrite 0x03b8ac thru 0x03ffff (0x4754 bytes) 000521 1733397508.627787 mapread 0x014c07 thru 0x0249f3 (0xfded bytes) 000522 1733397508.632456 trunc from 0x040000 to 0x035295 (0xad6a bytes) 000523 1733397508.635967 write 0x001059 thru 0x006bf6 (0x5b9e bytes) 000524 1733397508.636493 write 0x01203d thru 0x012065 (0x29 bytes) 000526 1733397508.637069 write 0x023925 thru 0x032bf1 (0xf2cd bytes) truncating to largest ever: 0x3f76d 000527 1733397508.638950 trunc from 0x035296 to 0x03f76c (0xa4d7 bytes) 000528 1733397508.643847 MIRROR_RESYNC 6 mirrors 000530 1733397508.817402 mapread 0x013350 thru 0x014746 (0x13f7 bytes) 000532 1733397508.820132 mapwrite 0x01cc5e thru 0x022531 (0x58d4 bytes) 000533 1733397508.862996 mapread 0x013201 thru 0x019db5 (0x6bb5 bytes) 000534 1733397508.865787 write 0x02a320 thru 0x031a3c (0x771d bytes) 000535 1733397508.868427 mapread 0x0017a4 thru 0x002c1b (0x1478 bytes) 000538 1733397508.871540 write 0x0284ca thru 0x02a52c (0x2063 bytes) 000540 1733397508.873436 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000541 1733397508.920908 read 0x01f231 thru 0x0240fb (0x4ecb bytes) 000544 1733397508.923479 write 0x02972b thru 0x02b946 (0x221c bytes) 000545 1733397508.925716 trunc from 0x03f76d to 0x01a521 (0x2524b bytes) 000546 1733397508.932798 mapread 0x011ad6 thru 0x0163ce (0x48f9 bytes) 000547 1733397508.935275 mapread 0x003e15 thru 0x0128fa (0xeae6 bytes) 000551 1733397508.939873 mapread 0x00ef1e thru 0x0124fd (0x35e0 bytes) 000553 1733397508.940506 trunc from 0x01a522 to 0x00c016 (0xe50b bytes) 000554 1733397508.944971 MIRROR_RESYNC 5 mirrors 000555 1733397509.083015 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000556 1733397509.136167 mapwrite 0x026413 thru 0x02f2e7 (0x8ed5 bytes) 000558 1733397509.168928 mapwrite 0x00dc3c thru 0x01333d (0x5702 bytes) 000559 1733397509.182480 mapwrite 0x02eabd thru 0x03cb36 (0xe07a bytes) 000561 1733397509.203250 trunc from 0x03cb37 to 0x01fd44 (0x1cdf2 bytes) 000562 1733397509.207157 trunc from 0x01fd45 to 0x014185 (0xbbbf bytes) 000563 1733397509.210278 write 0x000201 thru 0x00e405 (0xe205 bytes) 000564 1733397509.213244 write 0x00dc58 thru 0x0121c8 (0x4571 bytes) 000565 1733397509.213810 write 0x005337 thru 0x014846 (0xf510 bytes) 000566 1733397509.214550 read 0x0111d0 thru 0x014846 (0x3677 bytes) 000570 1733397509.215444 mapread 0x0125a5 thru 0x014846 (0x22a2 bytes) 000571 1733397509.216954 MIRROR_EXTEND to 5 mirrors 000572 1733397509.315235 mapread 0x00ace6 thru 0x014846 (0x9b61 bytes) 000573 1733397509.322523 MIRROR_RESYNC 5 mirrors 000574 1733397509.442337 mapwrite 0x003ec5 thru 0x00dd29 (0x9e65 bytes) 000575 1733397509.484533 trunc from 0x014847 to 0x01b3bc (0x6b76 bytes) 000576 1733397509.492654 trunc from 0x01b3bd to 0x03583b (0x1a47f bytes) 000578 1733397509.496267 write 0x001304 thru 0x010186 (0xee83 bytes) 000581 1733397509.497395 mapread 0x028a78 thru 0x03583b (0xcdc4 bytes) 000582 1733397509.501979 mapread 0x006f02 thru 0x008ded (0x1eec bytes) 000586 1733397509.503430 read 0x02a678 thru 0x02e028 (0x39b1 bytes) 000587 1733397509.503725 mapread 0x0339dd thru 0x03583b (0x1e5f bytes) 000589 1733397509.504276 read 0x02c140 thru 0x02e39f (0x2260 bytes) 000590 1733397509.504535 write 0x0167af thru 0x01ad6f (0x45c1 bytes) 000593 1733397509.508031 read 0x00332e thru 0x00f3f1 (0xc0c4 bytes) 000594 1733397509.510149 MIRROR_EXTEND to 6 mirrors 000595 1733397509.605572 mapread 0x029ed3 thru 0x02faea (0x5c18 bytes) 000596 1733397509.611930 trunc from 0x03583c to 0x034c48 (0xbf3 bytes) 000599 1733397509.633256 MIRROR_RESYNC 6 mirrors 000600 1733397509.789332 mapread 0x01e7f9 thru 0x02bcc8 (0xd4d0 bytes) 000601 1733397509.795009 mapwrite 0x01d20a thru 0x02829b (0xb092 bytes) 000604 1733397509.835836 trunc from 0x034c49 to 0x0004a3 (0x347a5 bytes) 000606 1733397509.845144 MIRROR_EXTEND to 7 mirrors 000607 1733397509.926640 mapread 0x000060 thru 0x0004a3 (0x444 bytes) 000609 1733397509.929661 write 0x0155f8 thru 0x01fbb0 (0xa5b9 bytes) 000610 1733397509.943183 read 0x0098c2 thru 0x0152e4 (0xba23 bytes) 000611 1733397509.946903 MIRROR_RESYNC 7 mirrors 000612 1733397510.123164 mapread 0x0085b1 thru 0x008ad3 (0x523 bytes) 000613 1733397510.125599 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000614 1733397510.185463 read 0x0078d1 thru 0x0164be (0xebee bytes) 000615 1733397510.188242 read 0x01f2e1 thru 0x01fbb0 (0x8d0 bytes) 000619 1733397510.190927 read 0x011f23 thru 0x01d3b8 (0xb496 bytes) 000620 1733397510.191208 write 0x00295d thru 0x010416 (0xdaba bytes) 000621 1733397510.207592 trunc from 0x01fbb1 to 0x001302 (0x1e8ae bytes) 000622 1733397510.213181 MIRROR_RESYNC 6 mirrors 000626 1733397510.364856 mapwrite 0x00b4e9 thru 0x01a7b3 (0xf2cb bytes) 000627 1733397510.402373 MIRROR_RESYNC 6 mirrors 000628 1733397510.558605 write 0x020b0e thru 0x02667e (0x5b71 bytes) 000634 1733397510.574885 read 0x004548 thru 0x006373 (0x1e2c bytes) 000635 1733397510.577149 write 0x01a28a thru 0x0226f4 (0x846b bytes) 000637 1733397510.579661 write 0x024fea thru 0x030354 (0xb36b bytes) 000638 1733397510.580709 read 0x01cb90 thru 0x021094 (0x4505 bytes) 000639 1733397510.581158 mapread 0x017b23 thru 0x01e493 (0x6971 bytes) 000640 1733397510.584519 mapread 0x0209fd thru 0x026f87 (0x658b bytes) 000641 1733397510.585205 mapwrite 0x0264a5 thru 0x030f70 (0xaacc bytes) 000642 1733397510.609855 mapwrite 0x020a6b thru 0x02309f (0x2635 bytes) 000645 1733397510.622981 read 0x00d0f4 thru 0x014408 (0x7315 bytes) 000648 1733397510.623833 write 0x00fc93 thru 0x01c9e4 (0xcd52 bytes) 000649 1733397510.626016 MIRROR_RESYNC 6 mirrors 000652 1733397510.797844 trunc from 0x030f71 to 0x021835 (0xf73b bytes) 000653 1733397510.815265 read 0x006248 thru 0x00acce (0x4a87 bytes) 000654 1733397510.817456 trunc from 0x021836 to 0x00a1fa (0x1763b bytes) 000655 1733397510.822199 mapwrite 0x012e35 thru 0x01fc94 (0xce60 bytes) 000657 1733397510.846483 MIRROR_EXTEND to 7 mirrors 000659 1733397510.933796 mapread 0x019d1e thru 0x01dade (0x3dc1 bytes) 000662 1733397510.943277 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000663 1733397511.003829 mapread 0x01625c thru 0x01909e (0x2e43 bytes) 000664 1733397511.014753 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000665 1733397511.075201 read 0x003f4b thru 0x008525 (0x45db bytes) 000667 1733397511.081057 mapread 0x0052fc thru 0x0121d8 (0xcedd bytes) 000668 1733397511.088121 trunc from 0x01fc95 to 0x002392 (0x1d902 bytes) 000669 1733397511.105667 trunc from 0x002393 to 0x00de79 (0xbae7 bytes) 000672 1733397511.110390 mapwrite 0x02b46f thru 0x035bdc (0xa76e bytes) 000673 1733397511.129839 mapwrite 0x00d374 thru 0x010096 (0x2d23 bytes) 000674 1733397511.142928 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000677 1733397511.186079 mapwrite 0x011f44 thru 0x01b9b2 (0x9a6f bytes) 000679 1733397511.201846 read 0x0029fa thru 0x00efe2 (0xc5e9 bytes) 000680 1733397511.204257 mapread 0x01fc0a thru 0x026b13 (0x6f0a bytes) 000681 1733397511.206629 write 0x0102ee thru 0x01d9a4 (0xd6b7 bytes) 000682 1733397511.209669 MIRROR_RESYNC 4 mirrors 000683 1733397511.337361 mapwrite 0x031ee8 thru 0x032f4b (0x1064 bytes) 000684 1733397511.379134 mapwrite 0x035809 thru 0x03ffff (0xa7f7 bytes) 000686 1733397511.403372 mapwrite 0x0326d1 thru 0x036669 (0x3f99 bytes) 000687 1733397511.414356 mapread 0x00a634 thru 0x00eb72 (0x453f bytes) 000688 1733397511.418184 MIRROR_RESYNC 4 mirrors 000689 1733397511.531581 read 0x03cc6a thru 0x03ffff (0x3396 bytes) 000691 1733397511.533637 mapread 0x02946f thru 0x038928 (0xf4ba bytes) 000693 1733397511.540229 mapread 0x0236f9 thru 0x029fa3 (0x68ab bytes) 000695 1733397511.542670 mapwrite 0x0042e4 thru 0x006324 (0x2041 bytes) 000696 1733397511.586066 mapread 0x0045f5 thru 0x00932d (0x4d39 bytes) 000697 1733397511.586746 write 0x0350cf thru 0x039cf9 (0x4c2b bytes) 000698 1733397511.589805 mapread 0x03a0a4 thru 0x03ffff (0x5f5c bytes) 000699 1733397511.591522 trunc from 0x040000 to 0x02007b (0x1ff84 bytes) 000700 1733397511.600406 mapread 0x01d893 thru 0x02007b (0x27e9 bytes) 000701 1733397511.604445 MIRROR_EXTEND to 5 mirrors 000702 1733397511.693709 mapread 0x016572 thru 0x019172 (0x2c01 bytes) 000703 1733397511.699908 write 0x016dcd thru 0x025159 (0xe38d bytes) 000704 1733397511.716026 mapwrite 0x005bf0 thru 0x00847b (0x288c bytes) 000705 1733397511.735704 MIRROR_EXTEND to 6 mirrors 000707 1733397511.831771 trunc from 0x02515a to 0x032627 (0xd4ce bytes) 000708 1733397511.852655 mapwrite 0x03e359 thru 0x03ffff (0x1ca7 bytes) 000710 1733397511.872173 trunc from 0x040000 to 0x015d09 (0x2a2f6 bytes) 000711 1733397511.876216 write 0x02954b thru 0x02bac2 (0x2578 bytes) 000712 1733397511.878088 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000713 1733397511.924678 MIRROR_EXTEND to 6 mirrors 000714 1733397512.016474 read 0x01a9f1 thru 0x021521 (0x6b31 bytes) 000715 1733397512.019443 write 0x009c3a thru 0x015feb (0xc3b2 bytes) 000716 1733397512.035503 mapread 0x0109f2 thru 0x013f2f (0x353e bytes) 000717 1733397512.036088 trunc from 0x02bac3 to 0x034253 (0x8791 bytes) 000719 1733397512.044192 write 0x007a88 thru 0x00af22 (0x349b bytes) 000720 1733397512.046620 mapwrite 0x00b69d thru 0x011e2a (0x678e bytes) 000721 1733397512.061476 read 0x023084 thru 0x030c7a (0xdbf7 bytes) 000722 1733397512.064985 trunc from 0x034254 to 0x020044 (0x1420f bytes) 000723 1733397512.070151 MIRROR_RESYNC 6 mirrors 000724 1733397512.237705 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000725 1733397512.294484 mapread 0x000ec9 thru 0x0038f3 (0x2a2b bytes) 000727 1733397512.299876 mapread 0x0009fd thru 0x008df6 (0x83fa bytes) 000728 1733397512.300225 mapwrite 0x022a67 thru 0x031238 (0xe7d2 bytes) 000729 1733397512.336091 MIRROR_EXTEND to 6 mirrors 000730 1733397512.424657 mapwrite 0x039d96 thru 0x03ffff (0x626a bytes) 000731 1733397512.459508 read 0x02c00c thru 0x037614 (0xb609 bytes) 000733 1733397512.462145 trunc from 0x040000 to 0x01634c (0x29cb3 bytes) 000734 1733397512.466751 write 0x01ec91 thru 0x01f4b7 (0x827 bytes) 000735 1733397512.467230 trunc from 0x01f4b8 to 0x02ce04 (0xd94d bytes) 000736 1733397512.470432 write 0x009788 thru 0x01699a (0xd213 bytes) 000737 1733397512.475473 MIRROR_EXTEND to 7 mirrors 000740 1733397512.577627 MIRROR_RESYNC 7 mirrors 000743 1733397512.742080 trunc from 0x02ce05 to 0x00acd3 (0x22131 bytes) 000747 1733397512.759426 read 0x001559 thru 0x0062b5 (0x4d5d bytes) 000749 1733397512.762151 trunc from 0x00acd4 to 0x008866 (0x246d bytes) 000750 1733397512.766996 mapwrite 0x03a13a thru 0x03cb60 (0x2a27 bytes) 000753 1733397512.786088 read 0x0043a6 thru 0x013db3 (0xfa0e bytes) 000754 1733397512.789699 mapread 0x01385d thru 0x01ed07 (0xb4ab bytes) 000755 1733397512.790122 read 0x028963 thru 0x02fdc0 (0x745e bytes) 000756 1733397512.790483 mapwrite 0x0259b9 thru 0x02db4b (0x8193 bytes) 000759 1733397512.802526 read 0x012a4d thru 0x016c2c (0x41e0 bytes) 000760 1733397512.804387 MIRROR_EXTEND to 8 mirrors 000761 1733397512.898353 mapwrite 0x01657e thru 0x01efbc (0x8a3f bytes) 000763 1733397512.941733 read 0x026864 thru 0x029cae (0x344b bytes) 000764 1733397512.943283 mapread 0x006784 thru 0x015ccd (0xf54a bytes) 000765 1733397512.947100 mapread 0x02da06 thru 0x032c70 (0x526b bytes) 000766 1733397512.949509 mapread 0x02ecfd thru 0x03bd60 (0xd064 bytes) 000769 1733397512.951638 trunc from 0x03cb61 to 0x00b284 (0x318dc bytes) 000772 1733397512.959930 trunc from 0x00b285 to 0x013179 (0x7ef5 bytes) 000774 1733397512.963323 mapread 0x00850f thru 0x00b987 (0x3479 bytes) 000775 1733397512.963760 mapwrite 0x00a145 thru 0x00d086 (0x2f42 bytes) 000776 1733397512.978323 MIRROR_RESYNC 8 mirrors 000778 1733397513.173655 read 0x004caa thru 0x010797 (0xbaee bytes) 000779 1733397513.175941 mapwrite 0x0057c9 thru 0x01344c (0xdc84 bytes) 000780 1733397513.209589 mapwrite 0x00600a thru 0x0095ea (0x35e1 bytes) 000781 1733397513.218895 write 0x0381e1 thru 0x03ffff (0x7e1f bytes) 000784 1733397513.221271 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000786 1733397513.266287 read 0x032ff3 thru 0x03b350 (0x835e bytes) 000787 1733397513.268699 trunc from 0x040000 to 0x017d24 (0x282db bytes) 000788 1733397513.273358 MIRROR_RESYNC 7 mirrors 000789 1733397513.435934 trunc from 0x017d25 to 0x032914 (0x1abf0 bytes) 000791 1733397513.454822 mapwrite 0x00bac9 thru 0x015b46 (0xa07e bytes) 000793 1733397513.473445 mapread 0x0192c6 thru 0x01e70f (0x544a bytes) 000794 1733397513.476225 mapread 0x01e1f1 thru 0x025226 (0x7036 bytes) 000797 1733397513.478559 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000799 1733397513.522815 write 0x038d07 thru 0x03ffff (0x72f9 bytes) 000800 1733397513.523503 write 0x027aec thru 0x02b415 (0x392a bytes) 000801 1733397513.526017 read 0x039aa5 thru 0x03ffff (0x655b bytes) 000803 1733397513.526767 mapwrite 0x00867d thru 0x014451 (0xbdd5 bytes) 000806 1733397513.545129 mapread 0x02b7a8 thru 0x02fdee (0x4647 bytes) 000807 1733397513.548005 trunc from 0x040000 to 0x024e60 (0x1b19f bytes) 000808 1733397513.551664 mapread 0x023ee3 thru 0x024e60 (0xf7e bytes) 000810 1733397513.554738 write 0x01b37d thru 0x020a1e (0x56a2 bytes) 000813 1733397513.556899 MIRROR_EXTEND to 7 mirrors 000818 1733397513.665914 mapwrite 0x010075 thru 0x014e70 (0x4dfc bytes) 000820 1733397513.709934 write 0x000cf7 thru 0x00255f (0x1869 bytes) 000824 1733397513.713223 write 0x034966 thru 0x0397f1 (0x4e8c bytes) 000825 1733397513.713808 write 0x0393fe thru 0x03ffff (0x6c02 bytes) 000826 1733397513.715633 MIRROR_EXTEND to 8 mirrors 000827 1733397513.812388 write 0x023128 thru 0x02c6b5 (0x958e bytes) 000828 1733397513.827495 mapread 0x015f70 thru 0x01fbd7 (0x9c68 bytes) 000829 1733397513.832771 MIRROR_RESYNC 8 mirrors 000830 1733397513.997950 MIRROR_EXTEND to 9 mirrors 000832 1733397514.118660 read 0x005c2a thru 0x005e7f (0x256 bytes) truncating to largest ever: 0x3fb82 000833 1733397514.120106 trunc from 0x040000 to 0x03fb81 (0x47e bytes) 000836 1733397514.137811 trunc from 0x03fb82 to 0x0266dc (0x194a5 bytes) 000837 1733397514.142690 mapwrite 0x00b652 thru 0x00f38f (0x3d3e bytes) 000838 1733397514.159605 read 0x019e88 thru 0x02497d (0xaaf6 bytes) 000839 1733397514.162286 mapread 0x025b79 thru 0x0266dc (0xb64 bytes) 000842 1733397514.164821 MIRROR_EXTEND to 10 mirrors 000845 1733397514.255282 read 0x005568 thru 0x00d362 (0x7dfb bytes) 000847 1733397514.259251 read 0x016645 thru 0x020741 (0xa0fd bytes) 000850 1733397514.266003 MIRROR_RESYNC 10 mirrors 000852 1733397514.484632 mapread 0x0022ad thru 0x004a01 (0x2755 bytes) 000857 1733397514.489548 mapwrite 0x029ba0 thru 0x03762f (0xda90 bytes) 000859 1733397514.527510 write 0x0025dd thru 0x004e23 (0x2847 bytes) 000866 1733397514.531897 mapread 0x032068 thru 0x03762f (0x55c8 bytes) 000868 1733397514.532738 write 0x02ba0a thru 0x02e30c (0x2903 bytes) 000869 1733397514.533373 trunc from 0x037630 to 0x034cb4 (0x297b bytes) 000870 1733397514.538009 read 0x003105 thru 0x010fac (0xdea8 bytes) 000875 1733397514.539690 mapwrite 0x00d9c1 thru 0x014b08 (0x7148 bytes) 000878 1733397514.556207 trunc from 0x034cb5 to 0x02843b (0xc879 bytes) 000879 1733397514.560314 write 0x03ef50 thru 0x03ffff (0x10b0 bytes) 000880 1733397514.560815 mapwrite 0x012bca thru 0x01cad4 (0x9f0b bytes) 000881 1733397514.575952 mapwrite 0x0312fc thru 0x03ffff (0xed04 bytes) 000882 1733397514.592703 write 0x01c806 thru 0x027780 (0xaf7b bytes) 000883 1733397514.594646 MIRROR_EXTEND to 11 mirrors 000889 1733397514.712418 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000890 1733397514.775501 trunc from 0x040000 to 0x031ad4 (0xe52b bytes) 000891 1733397514.796533 MIRROR_EXTEND to 11 mirrors 000894 1733397514.891014 read 0x00750e thru 0x00a8fa (0x33ed bytes) 000895 1733397514.893312 write 0x036b10 thru 0x03ffa3 (0x9494 bytes) 000896 1733397514.906437 mapread 0x0133d6 thru 0x01cc7f (0x98aa bytes) 000897 1733397514.910749 mapwrite 0x0241d4 thru 0x027f21 (0x3d4e bytes) 000898 1733397514.925307 mapread 0x00f39c thru 0x01ca58 (0xd6bd bytes) 000899 1733397514.928080 trunc from 0x03ffa4 to 0x02a4b4 (0x15aef bytes) 000900 1733397514.936646 trunc from 0x02a4b5 to 0x02ad66 (0x8b2 bytes) 000902 1733397514.940024 mapread 0x00d0db thru 0x01afb0 (0xded6 bytes) 000903 1733397514.940607 mapread 0x01a7e2 thru 0x01e73d (0x3f5c bytes) 000904 1733397514.941034 mapwrite 0x011382 thru 0x016957 (0x55d6 bytes) 000905 1733397514.953242 write 0x0213a6 thru 0x0220ab (0xd06 bytes) 000906 1733397514.954972 MIRROR_EXTEND to 12 mirrors 000907 1733397515.058360 write 0x00340c thru 0x00cf06 (0x9afb bytes) 000909 1733397515.073284 read 0x016918 thru 0x017ef6 (0x15df bytes) 000910 1733397515.074790 mapread 0x01823a thru 0x022cfa (0xaac1 bytes) 000911 1733397515.078794 mapwrite 0x020cb8 thru 0x02dfbd (0xd306 bytes) 000913 1733397515.099808 mapread 0x01ae09 thru 0x0253bb (0xa5b3 bytes) 000914 1733397515.102116 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000915 1733397515.148499 mapread 0x01f316 thru 0x029838 (0xa523 bytes) 000918 1733397515.155114 write 0x028fc6 thru 0x02e508 (0x5543 bytes) 000919 1733397515.155677 read 0x0120bc thru 0x021bc8 (0xfb0d bytes) 000920 1733397515.157626 mapwrite 0x00de17 thru 0x010f3e (0x3128 bytes) 000921 1733397515.171456 mapread 0x01867f thru 0x027c61 (0xf5e3 bytes) 000923 1733397515.173542 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000924 1733397515.219463 mapread 0x0010b1 thru 0x00c83c (0xb78c bytes) 000926 1733397515.225783 write 0x01783d thru 0x021b5e (0xa322 bytes) 000927 1733397515.229469 mapwrite 0x03244f thru 0x034dec (0x299e bytes) 000928 1733397515.249126 write 0x03a812 thru 0x03ffff (0x57ee bytes) 000929 1733397515.249845 mapread 0x0186ec thru 0x020ef2 (0x8807 bytes) 000930 1733397515.250457 read 0x0176cf thru 0x01a377 (0x2ca9 bytes) 000937 1733397515.252009 write 0x02ea37 thru 0x0345e0 (0x5baa bytes) 000938 1733397515.252605 mapwrite 0x00f94d thru 0x016190 (0x6844 bytes) 000939 1733397515.265639 mapwrite 0x002b01 thru 0x012087 (0xf587 bytes) 000940 1733397515.278845 mapwrite 0x019fca thru 0x01ede9 (0x4e20 bytes) 000942 1733397515.288670 trunc from 0x040000 to 0x023dbd (0x1c242 bytes) 000943 1733397515.292124 write 0x01eac2 thru 0x02ba6e (0xcfad bytes) 000944 1733397515.294019 MIRROR_RESYNC 10 mirrors 000945 1733397515.536433 mapread 0x029057 thru 0x02ba6e (0x2a18 bytes) 000946 1733397515.539077 read 0x028f1b thru 0x02ba6e (0x2b54 bytes) 000950 1733397515.540901 mapwrite 0x01ef44 thru 0x022947 (0x3a04 bytes) 000951 1733397515.582154 mapread 0x018f83 thru 0x026411 (0xd48f bytes) 000955 1733397515.583369 read 0x018e33 thru 0x01a587 (0x1755 bytes) 000956 1733397515.583667 trunc from 0x02ba6f to 0x00d469 (0x1e605 bytes) 000957 1733397515.592004 write 0x00088d thru 0x006eb3 (0x6627 bytes) 000959 1733397515.595109 trunc from 0x00d46a to 0x02a3bd (0x1cf54 bytes) 000963 1733397515.599718 mapwrite 0x03a00f thru 0x03ac3a (0xc2c bytes) 000965 1733397515.618019 trunc from 0x03ac3b to 0x0182f8 (0x22942 bytes) 000967 1733397515.621892 write 0x00ae37 thru 0x00ebc0 (0x3d8a bytes) 000968 1733397515.623737 MIRROR_EXTEND to 11 mirrors 000969 1733397515.719117 trunc from 0x0182f9 to 0x037d3c (0x1fa44 bytes) 000970 1733397515.736058 read 0x0255a9 thru 0x028a29 (0x3481 bytes) 000971 1733397515.737848 trunc from 0x037d3d to 0x03b0da (0x339e bytes) 000972 1733397515.742022 trunc from 0x03b0db to 0x028baf (0x1252b bytes) 000974 1733397515.747491 MIRROR_RESYNC 11 mirrors 000975 1733397515.987657 mapwrite 0x01872f thru 0x022ab9 (0xa38b bytes) 000976 1733397516.031659 mapread 0x00b77d thru 0x011b88 (0x640c bytes) 000978 1733397516.034519 read 0x01384b thru 0x01d459 (0x9c0f bytes) 000979 1733397516.034877 mapread 0x0269bc thru 0x02739d (0x9e2 bytes) 000980 1733397516.036738 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x83d:0x0]: Invalid argument (22) 000983 1733397516.081372 mapread 0x00815c thru 0x0134ae (0xb353 bytes) 000984 1733397516.088003 mapwrite 0x03f78a thru 0x03ffff (0x876 bytes) 000985 1733397516.107825 write 0x002053 thru 0x00b470 (0x941e bytes) 000986 1733397516.108521 mapwrite 0x014f47 thru 0x0245a0 (0xf65a bytes) 000988 1733397516.126764 trunc from 0x040000 to 0x02abfd (0x15402 bytes) 000990 1733397516.130899 trunc from 0x02abfe to 0x017047 (0x13bb6 bytes) 000992 1733397516.134284 mapread 0x002043 thru 0x004690 (0x264e bytes) 000993 1733397516.134708 mapwrite 0x00d41d thru 0x0140d1 (0x6cb5 bytes) 000996 1733397516.148768 mapwrite 0x009fb7 thru 0x01560b (0xb655 bytes) 000997 1733397516.160985 MIRROR_RESYNC 10 mirrors 000998 1733397516.377427 trunc from 0x017048 to 0x03593e (0x1e8f7 bytes) 000999 1733397516.395132 trunc from 0x03593f to 0x007396 (0x2e5a8 bytes) 001000 1733397516.399752 mapwrite 0x016a9a thru 0x0211b9 (0xa720 bytes) All operations completed A-OK! PASS 70a (15s) == sanity-flr test 200a: stress test ===================== 06:18:37 (1733397517) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9137184 .. Extending file size to 50592 .. Extending file size to 5581376 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7258592 .. Extending file size to 7970176 .. Extending file size to 8438912 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9162688 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9347392 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8623936 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8903744 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 1907 1908 1909 1911 1914 failed 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) 192.168.202.133@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg233-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:19:40 (1733397580) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8351520 .. Extending file size to 4177824 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7359808 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8396288 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8510848 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9134048 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8754720 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8816032 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9040000 .. failed Extending file size to 9218208 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 5309 5310 5312 5314 5316 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) 192.168.202.133@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg233-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:20:44 (1733397644) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 5120 2203136 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 5120 2203136 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0308656 s, 67.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0740826 s, 28.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:20:48 (1733397648) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0202049 s, 104 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 06:20:50 (1733397650) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x88c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6c:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xd6e:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:20:51 (1733397651) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0279429 s, 75.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd70:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:20:53 (1733397653) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0247868 s, 84.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd71:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd71:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd72:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:20:55 (1733397655) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0216037 s, 97.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd70:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd73:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd70:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd73:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd74:0x0] } PASS 204c (22s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:21:17 (1733397677) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193023 s, 54.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd71:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd76:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd75:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd71:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd76:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd75:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd72:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:22:00 (1733397720) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd73:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd74:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0263876 s, 79.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd73:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd74:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd77:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd73:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd74:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd77:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:22:40 (1733397760) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. Creating new pool oleg233-server: Pool lustre.test_204f created Adding targets to pool oleg233-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg233-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0291168 s, 72.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd75:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd78:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd75:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd78:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg233-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg233-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg233-server: Pool lustre.test_204f destroyed PASS 204f (52s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:23:32 (1733397812) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:23:34 (1733397814) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:23:35 (1733397815) oleg233-server: Pool lustre.test_206 created Adding targets to pool oleg233-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg233-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg233-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg233-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg233-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:23:48 (1733397828) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0185785 s, 56.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00827576 s, 127 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xb3983e7f Mirror 3: 0xb3983e7f PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:23:50 (1733397830) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:23:51 (1733397831) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:23:52 (1733397832) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0171426 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1a60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1a60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1a60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1290000 Read this (4 bytes): BBBB 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:24:05 (1733397845) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00233346 s, 3.5 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:24:07 (1733397847) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101595 s, 103 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x8b1 lmm_fid: [0x200000404:0x8b1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x8b1 lmm_fid: [0x200000404:0x8b1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd82:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836145 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-05 06:24:07.000000000 -0500 Modify: 2024-12-05 06:24:07.000000000 -0500 Change: 2024-12-05 06:24:07.000000000 -0500 Birth: - PASS 210a (1s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:24:08 (1733397848) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test complete, duration 643 sec ============ 06:24:09 (1733397849) === sanity-flr: start cleanup 06:24:09 (1733397849) === === sanity-flr: finish cleanup 06:24:11 (1733397851) ===