-----============= acceptance-small: sanity-flr ============----- Fri Dec 20 00:00:11 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg430-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 00:00:24 (1734670824) === oleg430-client.virtnet: executing check_config_client /mnt/lustre oleg430-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg430-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b58de000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b58de000.idle_timeout=debug disable quota as required oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 00:00:46 (1734670846) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7652] == sanity-flr test 0a: lfs mirror create with -N option == 00:00:47 (1734670847) PASS 0a (9s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 00:00:56 (1734670856) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 00:00:58 (1734670858) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 00:01:00 (1734670860) PASS 0d (14s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 00:01:14 (1734670874) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 00:01:17 (1734670877) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 00:01:19 (1734670879) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b58de000.stats=clear osc.lustre-OST0001-osc-ffff8800b58de000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 00:01:24 (1734670884) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 00:01:30 (1734670890) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 00:01:36 (1734670896) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 00:01:38 (1734670898) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 00:01:44 (1734670904) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 00:01:48 (1734670908) PASS 3 (6s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 00:01:54 (1734670914) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 00:01:59 (1734670919) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 00:02:06 (1734670926) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 00:02:11 (1734670931) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.01417 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0686611 s, 15.3 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 00:02:18 (1734670938) 19+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.253983 s, 78.4 MB/s PASS 22 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 00:02:24 (1734670944) Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58de000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58de000.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b58de000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58de000.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg430-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (46s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 00:03:10 (1734670990) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.519845 s, 20.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58de000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58de000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (23s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 00:03:34 (1734671014) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58de000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58de000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b58de000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58de000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg430-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (42s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 00:04:17 (1734671057) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.88553 s, 27.8 MB/s -rw-r--r-- 1 root root 52428800 Dec 20 00:04 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.641s user 0m0.002s sys 0m0.146s Stopping /mnt/lustre-ost1 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b58de000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b58de000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.679s user 0m0.000s sys 0m0.132s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg430-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg430-server oleg430-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b58de000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58de000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.715s user 0m0.000s sys 0m0.152s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg430-server: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg430-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg430-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b58de000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b58de000.ost_server_uuid in FULL state after 0 sec PASS 33b (45s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 00:05:02 (1734671102) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 00:05:04 (1734671104) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 00:05:06 (1734671106) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 00:05:09 (1734671109) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 00:05:13 (1734671113) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.290285 s, 43.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 00:05:20 (1734671120) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.026596 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0191336 s, 54.8 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 00:05:25 (1734671125) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 00:05:30 (1734671130) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000634083 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00231261 s, 443 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00284988 s, 359 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00145236 s, 705 kB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 00:05:35 (1734671135) create test files with size 15730858 .. 3b19d2b7d80d350cee079aa92eebb996 - 2fc777b4dfc2e909881cd55e247d7b09 - e72a59cfdb5cbde7e42bbb669314fd61 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg430-client.virtnet /mnt/lustre (opts:) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre PASS 37 (13s) == sanity-flr test 38: resync ============================ 00:05:48 (1734671148) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 00:05:55 (1734671155) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 00:05:59 (1734671159) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0418279 s, 25.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x37:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023086 s, 45.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x38:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (6s) == sanity-flr test 41: lfs mirror resync check =========== 00:06:05 (1734671165) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0511829 s, 41.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.193646 s, 21.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.122788 s, 17.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (7s) == sanity-flr test 42: lfs mirror verify ================= 00:06:12 (1734671172) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 00:06:14 (1734671174) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 00:06:17 (1734671177) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0258448 s, 40.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 00:06:22 (1734671182) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.153505 s, 20.5 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (6s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 00:06:28 (1734671188) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x43:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 00:06:33 (1734671193) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.249257 s, 42.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg430-client.virtnet /mnt/lustre (opts:) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 00:06:41 (1734671201) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800b61a7800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.348347 s, 30.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg430-client.virtnet /mnt/lustre (opts:) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8801309c7000.xattr_cache=1 PASS 44e (11s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 00:06:52 (1734671212) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (19s) == sanity-flr test 46: Verify setstripe --copy option ==== 00:07:11 (1734671231) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (21s) == sanity-flr test 47: Verify mirror obj alloc =========== 00:07:32 (1734671252) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 00:07:35 (1734671255) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.180765 s, 17.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.129594 s, 24.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 45e461d82a61da34844f1b3fc072797f - mirror 1 checksum: d0064c51b528f4b4b54ac714a3392696 - mirror 2 checksum: 45e461d82a61da34844f1b3fc072797f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Dec 20 00:07:37 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Dec 20 00:07:37 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 45e461d82a61da34844f1b3fc072797f - mirror 1 checksum: d0064c51b528f4b4b54ac714a3392696 - mirror 2 checksum: d0064c51b528f4b4b54ac714a3392696 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Dec 20 00:07:37 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Dec 20 00:07:37 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } PASS 48 (6s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 00:07:41 (1734671261) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166732 s, 62.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 527360.. 528383: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (5s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 00:07:47 (1734671267) Failing mds1 on oleg430-server Stopping /mnt/lustre-mds1 (opts:) on oleg430-server 00:07:56 (1734671276) shut down facet: mds1 facet_host: oleg430-server facet_failover_host: oleg430-server Failover mds1 to oleg430-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0000 00:08:13 (1734671293) targets are mounted 00:08:13 (1734671293) facet_failover done oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156467 s, 67.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 50A (37s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 00:08:23 (1734671303) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0293314 s, 698 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0150128 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (6s) == sanity-flr test 50b: mirror rsync handles sparseness == 00:08:29 (1734671309) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0264859 s, 773 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00591312 s, 3.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 00:08:34 (1734671314) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.0107289 s, 1.5 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 00:08:39 (1734671319) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0417774 s, 490 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 00:08:44 (1734671324) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00731292 s, 560 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0282757 s, 145 kB/s PASS 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 00:08:49 (1734671329) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (12s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 00:09:01 (1734671341) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 00:09:23 (1734671363) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 00:09:42 (1734671382) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 00:10:02 (1734671402) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 00:10:07 (1734671407) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 26507 mirror split pid 26508 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7547: 26507 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 26508 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 00:11:11 (1734671471) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 1472 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000001 1734671472.222693 write 0x022d16 thru 0x031687 (0xe972 bytes) 000002 1734671472.232909 write 0x004770 thru 0x008fcf (0x4860 bytes) 000003 1734671472.239583 punch from 0x0089e0 to 0x00d186 (0x47a7 bytes) fallocating to largest ever: 0x3926d 000004 1734671472.248902 fallocate 0x034d39 thru 0x03926c (0x4534 bytes) 000006 1734671472.252424 mapwrite 0x036cf6 thru 0x03ce6f (0x617a bytes) 000007 1734671472.291063 punch from 0x001e35 to 0x0033b3 (0x157f bytes) 000008 1734671472.294811 read 0x025161 thru 0x025704 (0x5a4 bytes) 000009 1734671472.295279 trunc from 0x03ce70 to 0x031077 (0xbdf8 bytes) 000013 1734671472.313797 read 0x00de60 thru 0x015632 (0x77d3 bytes) 000014 1734671472.322653 punch from 0x02bb56 to 0x031077 (0x5522 bytes) 000015 1734671472.326849 mapwrite 0x0196c9 thru 0x021aff (0x8437 bytes) 000016 1734671472.341026 read 0x01b7ac thru 0x01e474 (0x2cc9 bytes) 000017 1734671472.341409 trunc from 0x031078 to 0x036b8a (0x5b13 bytes) 000018 1734671472.353982 mapread 0x035a0a thru 0x036b8a (0x1181 bytes) 000019 1734671472.361013 write 0x003b4f thru 0x00ec5b (0xb10d bytes) 000021 1734671472.368586 MIRROR_EXTEND to 2 mirrors 000022 1734671472.740181 mapwrite 0x005a49 thru 0x005f34 (0x4ec bytes) 000023 1734671472.801993 write 0x012cac thru 0x018982 (0x5cd7 bytes) 000024 1734671472.816040 mapread 0x00f451 thru 0x012341 (0x2ef1 bytes) 000026 1734671472.817457 fallocate 0x004405 thru 0x004d2e (0x92a bytes) 000027 1734671472.828977 trunc from 0x036b8b to 0x00115d (0x35a2d bytes) 000029 1734671472.868867 write 0x00c086 thru 0x0182d5 (0xc250 bytes) 000030 1734671472.869981 mapread 0x006df3 thru 0x00de0f (0x701d bytes) 000031 1734671472.886173 MIRROR_RESYNC 2 mirrors 000032 1734671473.221445 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000035 1734671473.320528 read 0x00c299 thru 0x0164f2 (0xa25a bytes) 000036 1734671473.335765 write 0x03e565 thru 0x03ffff (0x1a9b bytes) 000037 1734671473.355619 punch from 0x01c629 to 0x02941d (0xcdf5 bytes) 000038 1734671473.363308 trunc from 0x040000 to 0x0255a6 (0x1aa59 bytes) 000039 1734671473.389995 mapread 0x010b14 thru 0x01ab6a (0xa057 bytes) 000040 1734671473.419424 mapwrite 0x010995 thru 0x016b7a (0x61e6 bytes) 000041 1734671473.453358 punch from 0x0132ac to 0x01f4d8 (0xc22d bytes) 000042 1734671473.461884 write 0x01a1ec thru 0x0215e4 (0x73f9 bytes) 000043 1734671473.475362 punch from 0x01d704 to 0x01de6c (0x769 bytes) 000044 1734671473.496850 punch from 0x023cc8 to 0x0255a6 (0x18df bytes) 000048 1734671473.503666 mapread 0x01aaab thru 0x01d4dd (0x2a33 bytes) 000050 1734671473.511666 read 0x0154a8 thru 0x018bd2 (0x372b bytes) 000052 1734671473.512494 punch from 0x010bda to 0x01a25f (0x9686 bytes) 000053 1734671473.517220 mapwrite 0x026ffd thru 0x02c268 (0x526c bytes) 000054 1734671473.569455 fallocate 0x0023f8 thru 0x009a97 (0x76a0 bytes) 000055 1734671473.574153 trunc from 0x02c269 to 0x02bfb5 (0x2b3 bytes) 000057 1734671473.587608 MIRROR_EXTEND to 2 mirrors 000058 1734671473.974425 write 0x02dde7 thru 0x039c6d (0xbe87 bytes) 000059 1734671474.003261 trunc from 0x039c6e to 0x02e874 (0xb3f9 bytes) 000060 1734671474.021536 mapwrite 0x00e361 thru 0x016455 (0x80f5 bytes) 000061 1734671474.050916 fallocate 0x018da7 thru 0x019309 (0x563 bytes) 000062 1734671474.053891 read 0x029849 thru 0x02e874 (0x502c bytes) 000063 1734671474.057983 read 0x005c87 thru 0x01594a (0xfcc4 bytes) truncating to largest ever: 0x3dd3c 000064 1734671474.064877 trunc from 0x02e875 to 0x03dd3b (0xf4c7 bytes) 000066 1734671474.076914 mapread 0x00ebf8 thru 0x00fb8d (0xf96 bytes) 000068 1734671474.078886 mapwrite 0x03b8a4 thru 0x03ffff (0x475c bytes) 000069 1734671474.118698 MIRROR_EXTEND to 3 mirrors 000072 1734671474.657673 punch from 0x01851c to 0x01937e (0xe63 bytes) 000075 1734671474.688109 write 0x0000bc thru 0x004abf (0x4a04 bytes) 000078 1734671474.707403 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000079 1734671474.882405 read 0x014c45 thru 0x0249b5 (0xfd71 bytes) 000080 1734671474.891272 punch from 0x01ec99 to 0x02572a (0x6a92 bytes) 000081 1734671474.897122 fallocate 0x02fe69 thru 0x039550 (0x96e8 bytes) 000082 1734671474.901337 punch from 0x0102a6 to 0x01eca8 (0xea03 bytes) 000084 1734671474.907263 mapwrite 0x019599 thru 0x01a4e8 (0xf50 bytes) 000085 1734671474.933847 read 0x01391d thru 0x01ba4e (0x8132 bytes) 000086 1734671474.962702 trunc from 0x040000 to 0x00c787 (0x33878 bytes) 000088 1734671475.007543 trunc from 0x00c788 to 0x018c0b (0xc484 bytes) 000089 1734671475.023599 trunc from 0x018c0c to 0x03a938 (0x21d2d bytes) 000091 1734671475.036124 punch from 0x00b64e to 0x0127d3 (0x7186 bytes) 000092 1734671475.042254 punch from 0x01821a to 0x01ac46 (0x2a2d bytes) 000093 1734671475.045667 mapwrite 0x02b3f1 thru 0x02f3c8 (0x3fd8 bytes) 000094 1734671475.086663 write 0x010d6d thru 0x016b80 (0x5e14 bytes) 000095 1734671475.093513 read 0x02d9e4 thru 0x033da7 (0x63c4 bytes) 000096 1734671475.099186 write 0x033aaa thru 0x036ca8 (0x31ff bytes) 000097 1734671475.099889 write 0x000056 thru 0x00b334 (0xb2df bytes) 000098 1734671475.100789 punch from 0x02bd85 to 0x02cb9a (0xe16 bytes) 000101 1734671475.108045 mapread 0x026ab0 thru 0x03330e (0xc85f bytes) 000102 1734671475.114946 punch from 0x021dbf to 0x0276e4 (0x5926 bytes) 000103 1734671475.119623 fallocate 0x01ad36 thru 0x02524e (0xa519 bytes) 000104 1734671475.123369 punch from 0x005032 to 0x00e6e7 (0x96b6 bytes) 000105 1734671475.139942 mapwrite 0x01d49d thru 0x029d80 (0xc8e4 bytes) 000107 1734671475.214564 mapwrite 0x02cdc7 thru 0x0376f1 (0xa92b bytes) 000108 1734671475.242313 write 0x027167 thru 0x033a08 (0xc8a2 bytes) 000109 1734671475.250128 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000110 1734671475.365299 write 0x00ad4f thru 0x00f3a2 (0x4654 bytes) 000111 1734671475.380397 trunc from 0x03a939 to 0x01ead5 (0x1be63 bytes) 000112 1734671475.389971 read 0x0028a9 thru 0x0058f0 (0x3048 bytes) 000114 1734671475.394963 write 0x014a5b thru 0x0182e9 (0x388f bytes) 000115 1734671475.402287 mapread 0x009c06 thru 0x014845 (0xac40 bytes) 000116 1734671475.407260 fallocate 0x027387 thru 0x02d09b (0x5d15 bytes) 000119 1734671475.411914 punch from 0x0075bf to 0x0127d2 (0xb214 bytes) 000122 1734671475.424033 fallocate 0x00750a thru 0x013aac (0xc5a3 bytes) 000124 1734671475.431487 fallocate 0x00a2e8 thru 0x019555 (0xf26e bytes) 000125 1734671475.436404 trunc from 0x01ead6 to 0x00906d (0x15a68 bytes) 000126 1734671475.448081 fallocate 0x010cee thru 0x01c5c1 (0xb8d4 bytes) 000127 1734671475.452416 write 0x039902 thru 0x03ffff (0x66fe bytes) 000128 1734671475.453301 read 0x020f0b thru 0x02b957 (0xaa4d bytes) 000129 1734671475.465315 MIRROR_EXTEND to 2 mirrors 000130 1734671475.838987 fallocate 0x027121 thru 0x02eec8 (0x7da8 bytes) 000131 1734671475.874197 write 0x0175ec thru 0x0247bc (0xd1d1 bytes) 000133 1734671475.883358 punch from 0x034f85 to 0x0398f5 (0x4971 bytes) 000134 1734671475.888278 mapread 0x00f29d thru 0x014f55 (0x5cb9 bytes) 000136 1734671475.896373 punch from 0x001c7f to 0x010b05 (0xee87 bytes) 000138 1734671475.901399 fallocate 0x0288c2 thru 0x036d36 (0xe475 bytes) 000139 1734671475.907229 punch from 0x010604 to 0x01c60c (0xc009 bytes) 000140 1734671475.919674 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000142 1734671476.059577 read 0x00ed18 thru 0x01350e (0x47f7 bytes) 000143 1734671476.067077 mapread 0x03154d thru 0x03245b (0xf0f bytes) 000144 1734671476.075489 read 0x03d529 thru 0x03ffff (0x2ad7 bytes) 000145 1734671476.086016 punch from 0x000e0d to 0x001034 (0x228 bytes) 000146 1734671476.092179 trunc from 0x040000 to 0x03b051 (0x4fae bytes) 000147 1734671476.125313 punch from 0x028182 to 0x0326b6 (0xa535 bytes) 000148 1734671476.130768 mapread 0x028801 thru 0x036918 (0xe118 bytes) 000149 1734671476.157291 mapwrite 0x03aeba thru 0x03b6bf (0x806 bytes) 000150 1734671476.187340 mapread 0x01ff09 thru 0x0222ce (0x23c6 bytes) 000152 1734671476.188201 trunc from 0x03b6c0 to 0x01e901 (0x1cdbe bytes) 000154 1734671476.204099 read 0x017164 thru 0x01e901 (0x779e bytes) 000155 1734671476.223623 mapwrite 0x02b78d thru 0x033f5e (0x87d2 bytes) 000156 1734671476.299998 trunc from 0x033f5f to 0x038c67 (0x4d09 bytes) 000158 1734671476.323186 trunc from 0x038c68 to 0x019d06 (0x1ef61 bytes) 000159 1734671476.341416 punch from 0x0025c7 to 0x00b1df (0x8c19 bytes) 000160 1734671476.349048 write 0x008871 thru 0x00f464 (0x6bf4 bytes) 000161 1734671476.357247 read 0x0115d8 thru 0x019d06 (0x872f bytes) 000163 1734671476.358021 write 0x02da45 thru 0x038661 (0xac1d bytes) 000164 1734671476.359834 mapread 0x02d9c7 thru 0x032799 (0x4dd3 bytes) 000165 1734671476.360508 read 0x03038e thru 0x038661 (0x82d4 bytes) 000166 1734671476.364444 MIRROR_EXTEND to 2 mirrors 000169 1734671476.670652 trunc from 0x038662 to 0x03b60d (0x2fac bytes) 000170 1734671476.714659 read 0x02fa61 thru 0x03b60d (0xbbad bytes) 000171 1734671476.719621 punch from 0x013a6b to 0x01b003 (0x7599 bytes) 000172 1734671476.722821 punch from 0x02ec34 to 0x03300c (0x43d9 bytes) 000173 1734671476.728198 write 0x00c25b thru 0x01832c (0xc0d2 bytes) 000175 1734671476.740267 write 0x00bc5d thru 0x014d01 (0x90a5 bytes) 000176 1734671476.746216 write 0x00f7e5 thru 0x013f35 (0x4751 bytes) 000179 1734671476.748339 mapread 0x0390fb thru 0x03b60d (0x2513 bytes) 000180 1734671476.751939 write 0x000a54 thru 0x0050e0 (0x468d bytes) 000183 1734671476.761353 trunc from 0x03b60e to 0x02c82e (0xeddf bytes) 000185 1734671476.779847 trunc from 0x02c82f to 0x033a20 (0x71f2 bytes) 000186 1734671476.797811 MIRROR_RESYNC 2 mirrors 000187 1734671477.162405 mapread 0x006bef thru 0x00ab4f (0x3f61 bytes) 000188 1734671477.173296 mapwrite 0x0035d1 thru 0x0057a3 (0x21d3 bytes) 000190 1734671477.242817 write 0x03e089 thru 0x03ffff (0x1f77 bytes) 000191 1734671477.244525 mapread 0x011367 thru 0x018f82 (0x7c1c bytes) 000192 1734671477.252090 write 0x0051ce thru 0x00e490 (0x92c3 bytes) 000193 1734671477.256414 mapread 0x019d74 thru 0x021953 (0x7be0 bytes) 000194 1734671477.261770 trunc from 0x040000 to 0x01e7ce (0x21831 bytes) 000195 1734671477.286580 mapread 0x01288d thru 0x01a209 (0x797d bytes) 000196 1734671477.290468 fallocate 0x011692 thru 0x01faf7 (0xe466 bytes) 000197 1734671477.297454 write 0x0233b3 thru 0x03104b (0xdc99 bytes) 000199 1734671477.298967 trunc from 0x03104c to 0x032a82 (0x1a37 bytes) 000200 1734671477.328122 punch from 0x030796 to 0x032a82 (0x22ed bytes) 000201 1734671477.353170 mapread 0x0283cf thru 0x032a82 (0xa6b4 bytes) 000202 1734671477.363701 fallocate 0x02b4d5 thru 0x03059f (0x50cb bytes) 000203 1734671477.372413 punch from 0x01ac94 to 0x021937 (0x6ca4 bytes) 000204 1734671477.392618 MIRROR_EXTEND to 3 mirrors 000205 1734671477.816269 trunc from 0x032a83 to 0x0020bb (0x309c7 bytes) 000206 1734671477.885402 punch from 0x0019f1 to 0x0020bb (0x6cb bytes) 000207 1734671477.890446 read 0x001a85 thru 0x0020bb (0x637 bytes) fallocating to largest ever: 0x40000 000208 1734671477.895146 fallocate 0x03f46c thru 0x03ffff (0xb94 bytes) 000209 1734671477.898498 fallocate 0x039a7f thru 0x03fe94 (0x6416 bytes) 000210 1734671477.901713 mapread 0x00b606 thru 0x017625 (0xc020 bytes) 000212 1734671477.936125 MIRROR_RESYNC 3 mirrors 000213 1734671478.369827 fallocate 0x01dfb2 thru 0x02a488 (0xc4d7 bytes) 000214 1734671478.392259 mapwrite 0x016f45 thru 0x026937 (0xf9f3 bytes) 000215 1734671478.435472 trunc from 0x040000 to 0x0119ea (0x2e615 bytes) 000216 1734671478.450972 punch from 0x00d77b to 0x0119ea (0x4270 bytes) 000217 1734671478.454548 trunc from 0x0119eb to 0x01c2fe (0xa914 bytes) 000218 1734671478.465610 mapread 0x0163c5 thru 0x01a3ea (0x4026 bytes) 000219 1734671478.473482 mapread 0x008de9 thru 0x013c8c (0xaea4 bytes) 000220 1734671478.483785 read 0x00729a thru 0x00d722 (0x6489 bytes) 000221 1734671478.485681 punch from 0x001f4e to 0x007786 (0x5839 bytes) 000223 1734671478.489507 mapread 0x018922 thru 0x01c2fe (0x39dd bytes) 000225 1734671478.495511 MIRROR_EXTEND to 4 mirrors 000227 1734671478.807543 MIRROR_RESYNC 4 mirrors 000229 1734671479.122512 punch from 0x013c13 to 0x01c2fe (0x86ec bytes) 000232 1734671479.143121 mapread 0x00abdb thru 0x00f23e (0x4664 bytes) 000233 1734671479.155092 punch from 0x00b92d to 0x014900 (0x8fd4 bytes) 000236 1734671479.159450 trunc from 0x01c2ff to 0x031003 (0x14d05 bytes) 000237 1734671479.192717 read 0x017782 thru 0x01d633 (0x5eb2 bytes) 000238 1734671479.199051 mapwrite 0x0170fb thru 0x025b82 (0xea88 bytes) 000241 1734671479.237091 MIRROR_EXTEND to 5 mirrors 000242 1734671479.672254 write 0x012f92 thru 0x019a27 (0x6a96 bytes) 000243 1734671479.711629 trunc from 0x031004 to 0x00e4f0 (0x22b13 bytes) 000244 1734671479.750665 fallocate 0x014d1e thru 0x024c0f (0xfef2 bytes) 000245 1734671479.757481 read 0x00f295 thru 0x018127 (0x8e93 bytes) 000246 1734671479.771094 write 0x02c589 thru 0x0314b5 (0x4f2d bytes) 000247 1734671479.773250 trunc from 0x0314b6 to 0x00a8e8 (0x26bcd bytes) 000248 1734671479.794345 trunc from 0x00a8e9 to 0x035e45 (0x2b55d bytes) 000249 1734671479.810660 MIRROR_EXTEND to 6 mirrors 000250 1734671480.168667 MIRROR_RESYNC 6 mirrors 000251 1734671480.740341 mapwrite 0x03e8f4 thru 0x03ffff (0x170c bytes) 000252 1734671480.809531 fallocate 0x011ef9 thru 0x01a6ba (0x87c2 bytes) 000253 1734671480.818941 punch from 0x00a2b5 to 0x00f8cb (0x5617 bytes) 000254 1734671480.824906 fallocate 0x01c0c8 thru 0x02abed (0xeb26 bytes) 000255 1734671480.833437 mapread 0x03277c thru 0x037802 (0x5087 bytes) 000256 1734671480.842021 mapwrite 0x00c70a thru 0x014d6e (0x8665 bytes) 000257 1734671480.863217 punch from 0x02aa77 to 0x02b3ab (0x935 bytes) 000258 1734671480.868965 fallocate 0x02ef5f thru 0x03d83c (0xe8de bytes) 000259 1734671480.875883 MIRROR_EXTEND to 7 mirrors 000260 1734671481.346868 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000261 1734671481.575409 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000262 1734671481.850563 trunc from 0x040000 to 0x023146 (0x1ceb9 bytes) 000263 1734671481.913619 mapread 0x008705 thru 0x00d4ce (0x4dca bytes) 000264 1734671481.928099 fallocate 0x003ae2 thru 0x00cb94 (0x90b3 bytes) 000265 1734671481.932093 punch from 0x01e66a to 0x023146 (0x4add bytes) 000266 1734671481.936880 read 0x002cde thru 0x011105 (0xe428 bytes) 000267 1734671481.945075 trunc from 0x023147 to 0x01f540 (0x3c06 bytes) 000268 1734671481.958347 fallocate 0x02312a thru 0x0320d6 (0xefad bytes) 000269 1734671481.962685 fallocate 0x02790c thru 0x034f38 (0xd62d bytes) 000271 1734671481.970502 MIRROR_EXTEND to 6 mirrors 000272 1734671482.488136 write 0x038371 thru 0x03ffff (0x7c8f bytes) 000273 1734671482.538107 punch from 0x023444 to 0x023979 (0x536 bytes) 000274 1734671482.555388 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000277 1734671482.724447 punch from 0x01ecda to 0x02a43a (0xb761 bytes) 000278 1734671482.734443 write 0x015ff5 thru 0x01c655 (0x6661 bytes) 000279 1734671482.756601 trunc from 0x040000 to 0x0146bc (0x2b943 bytes) 000281 1734671482.770645 write 0x03bfc1 thru 0x03fc3e (0x3c7e bytes) 000282 1734671482.772083 mapwrite 0x012b0a thru 0x015700 (0x2bf7 bytes) 000283 1734671482.802087 mapwrite 0x033126 thru 0x03f22b (0xc106 bytes) 000284 1734671482.843619 mapwrite 0x038918 thru 0x03ffff (0x76e8 bytes) 000285 1734671482.886199 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000286 1734671483.021364 read 0x036857 thru 0x03e346 (0x7af0 bytes) 000287 1734671483.027444 read 0x001325 thru 0x0106a4 (0xf380 bytes) 000288 1734671483.035642 write 0x0208ef thru 0x022696 (0x1da8 bytes) 000289 1734671483.039819 write 0x024864 thru 0x02fe27 (0xb5c4 bytes) 000290 1734671483.048341 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000292 1734671483.212126 MIRROR_EXTEND to 4 mirrors 000293 1734671483.597161 read 0x00eadd thru 0x01b004 (0xc528 bytes) 000294 1734671483.607298 write 0x00b3fd thru 0x018f6f (0xdb73 bytes) 000295 1734671483.673093 MIRROR_EXTEND to 5 mirrors 000297 1734671484.147849 read 0x00ddc1 thru 0x00e7d1 (0xa11 bytes) 000300 1734671484.169193 trunc from 0x040000 to 0x0392d2 (0x6d2d bytes) 000301 1734671484.236618 write 0x0223c9 thru 0x0290a0 (0x6cd8 bytes) 000303 1734671484.252340 read 0x01ecf2 thru 0x02d8f3 (0xec02 bytes) 000304 1734671484.263136 fallocate 0x00a70f thru 0x00f12e (0x4a20 bytes) 000305 1734671484.269412 trunc from 0x0392d3 to 0x0179cb (0x21907 bytes) 000306 1734671484.301406 mapread 0x00529a thru 0x00c58e (0x72f5 bytes) 000307 1734671484.302263 read 0x00a511 thru 0x0179cb (0xd4bb bytes) 000308 1734671484.307563 mapread 0x00f488 thru 0x0179cb (0x8544 bytes) 000309 1734671484.308229 read 0x004d22 thru 0x00681b (0x1afa bytes) 000310 1734671484.308564 mapread 0x006e45 thru 0x00fa86 (0x8c42 bytes) 000311 1734671484.309133 trunc from 0x0179cc to 0x031468 (0x19a9d bytes) 000314 1734671484.339178 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000316 1734671484.579705 punch from 0x004a67 to 0x00de03 (0x939d bytes) 000317 1734671484.588662 punch from 0x0313d6 to 0x031468 (0x93 bytes) 000318 1734671484.593072 write 0x003fff thru 0x00fee4 (0xbee6 bytes) 000319 1734671484.611811 write 0x00846b thru 0x0161aa (0xdd40 bytes) 000321 1734671484.619021 punch from 0x0213ee to 0x02d6ea (0xc2fd bytes) 000322 1734671484.623259 write 0x03655c thru 0x03d8b6 (0x735b bytes) 000323 1734671484.624707 write 0x01e4a8 thru 0x023d6c (0x58c5 bytes) 000325 1734671484.633301 read 0x016ff1 thru 0x01b063 (0x4073 bytes) 000326 1734671484.633961 fallocate 0x032361 thru 0x03c560 (0xa200 bytes) 000329 1734671484.645851 read 0x0333b1 thru 0x035bff (0x284f bytes) 000330 1734671484.651277 mapwrite 0x02ed6b thru 0x039d72 (0xb008 bytes) 000333 1734671484.681939 mapwrite 0x00b560 thru 0x013691 (0x8132 bytes) 000334 1734671484.697993 write 0x021f16 thru 0x026400 (0x44eb bytes) 000335 1734671484.698926 punch from 0x01a47c to 0x0217f3 (0x7378 bytes) 000338 1734671484.712615 trunc from 0x03d8b7 to 0x022e21 (0x1aa95 bytes) 000339 1734671484.732042 MIRROR_RESYNC 4 mirrors 000340 1734671485.332807 punch from 0x015fcb to 0x017068 (0x109e bytes) 000341 1734671485.381788 MIRROR_RESYNC 4 mirrors 000342 1734671486.072267 MIRROR_RESYNC 4 mirrors 000343 1734671486.196652 fallocate 0x01d5dc thru 0x02bf51 (0xe976 bytes) 000344 1734671486.230248 mapread 0x005d5f thru 0x00e338 (0x85da bytes) 000345 1734671486.242629 write 0x01d86c thru 0x02ad4d (0xd4e2 bytes) 000346 1734671486.253258 write 0x022236 thru 0x022c8a (0xa55 bytes) 000347 1734671486.257828 MIRROR_RESYNC 4 mirrors 000349 1734671486.930174 punch from 0x0192fe to 0x027a61 (0xe764 bytes) 000351 1734671486.955313 fallocate 0x0173a4 thru 0x01b52d (0x418a bytes) 000352 1734671486.962122 write 0x032e13 thru 0x036b6a (0x3d58 bytes) 000353 1734671486.963227 write 0x031749 thru 0x031880 (0x138 bytes) 000354 1734671486.973515 read 0x020d57 thru 0x027a86 (0x6d30 bytes) 000355 1734671486.982659 mapwrite 0x01dced thru 0x025c2b (0x7f3f bytes) 000357 1734671487.013671 mapread 0x008d9a thru 0x0105b8 (0x781f bytes) 000358 1734671487.028817 MIRROR_EXTEND to 5 mirrors 000359 1734671487.468182 punch from 0x019739 to 0x023655 (0x9f1d bytes) 000360 1734671487.518212 MIRROR_EXTEND to 6 mirrors 000361 1734671487.970211 fallocate 0x010e66 thru 0x01e5b1 (0xd74c bytes) 000362 1734671488.010933 fallocate 0x00d681 thru 0x01b417 (0xdd97 bytes) 000364 1734671488.016315 read 0x034f0e thru 0x036b6a (0x1c5d bytes) 000365 1734671488.021896 write 0x032647 thru 0x035d93 (0x374d bytes) 000366 1734671488.029789 trunc from 0x036b6b to 0x01c1f7 (0x1a973 bytes) 000367 1734671488.047916 trunc from 0x01c1f8 to 0x0349ec (0x187f5 bytes) 000369 1734671488.060848 mapwrite 0x003371 thru 0x003f09 (0xb99 bytes) 000370 1734671488.079998 mapwrite 0x01337b thru 0x01e270 (0xaef6 bytes) 000371 1734671488.121666 fallocate 0x00ea7c thru 0x01d3d7 (0xe95c bytes) 000372 1734671488.131149 punch from 0x026448 to 0x02da13 (0x75cc bytes) 000373 1734671488.139575 MIRROR_RESYNC 6 mirrors 000374 1734671489.017912 read 0x011f90 thru 0x01b6d0 (0x9741 bytes) 000376 1734671489.026579 punch from 0x03114d to 0x0349ec (0x38a0 bytes) 000377 1734671489.050316 fallocate 0x0384f0 thru 0x03ffff (0x7b10 bytes) 000380 1734671489.059515 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000381 1734671489.223667 punch from 0x0122ef to 0x01e334 (0xc046 bytes) 000382 1734671489.230617 trunc from 0x0349ed to 0x01da65 (0x16f87 bytes) 000383 1734671489.262033 trunc from 0x01da66 to 0x03e035 (0x205d0 bytes) 000384 1734671489.275251 trunc from 0x03e036 to 0x01f6f3 (0x1e942 bytes) 000385 1734671489.281777 mapread 0x00b722 thru 0x013510 (0x7def bytes) 000386 1734671489.288615 punch from 0x00afaa to 0x01a192 (0xf1e9 bytes) 000387 1734671489.291639 read 0x018d01 thru 0x019ca7 (0xfa7 bytes) 000388 1734671489.296013 mapwrite 0x03b1b0 thru 0x03ffff (0x4e50 bytes) 000390 1734671489.333606 trunc from 0x040000 to 0x034643 (0xb9bc bytes) 000394 1734671489.344263 punch from 0x005b33 to 0x0157da (0xfca8 bytes) 000395 1734671489.347939 punch from 0x01c083 to 0x029d36 (0xdcb4 bytes) 000396 1734671489.351908 punch from 0x0117bb to 0x01a78b (0x8fd1 bytes) 000397 1734671489.357456 read 0x00eb11 thru 0x018ad8 (0x9fc8 bytes) 000398 1734671489.366985 punch from 0x003f84 to 0x00734b (0x33c8 bytes) 000399 1734671489.376174 mapwrite 0x01bcf4 thru 0x024396 (0x86a3 bytes) 000400 1734671489.396525 read 0x02dfdd thru 0x034643 (0x6667 bytes) 000401 1734671489.396978 write 0x023a9e thru 0x028ca3 (0x5206 bytes) 000402 1734671489.397805 mapread 0x011522 thru 0x015b0e (0x45ed bytes) 000404 1734671489.398558 mapwrite 0x0095c0 thru 0x0139e4 (0xa425 bytes) 000405 1734671489.414616 write 0x01e00d thru 0x02b5b5 (0xd5a9 bytes) 000406 1734671489.415722 trunc from 0x034644 to 0x00f058 (0x255eb bytes) 000407 1734671489.427394 read 0x005f1e thru 0x00f058 (0x913b bytes) 000408 1734671489.432976 mapwrite 0x037d84 thru 0x03ffff (0x827c bytes) 000409 1734671489.472680 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000410 1734671489.609923 write 0x00bebf thru 0x0185df (0xc721 bytes) 000411 1734671489.621835 fallocate 0x001cdd thru 0x003910 (0x1c34 bytes) 000412 1734671489.626571 read 0x026469 thru 0x02d35d (0x6ef5 bytes) 000413 1734671489.633058 write 0x0343b3 thru 0x038e7b (0x4ac9 bytes) 000415 1734671489.638861 read 0x03b740 thru 0x03f024 (0x38e5 bytes) 000416 1734671489.646992 trunc from 0x040000 to 0x003662 (0x3c99d bytes) 000417 1734671489.657887 punch from 0x000216 to 0x003662 (0x344d bytes) 000419 1734671489.663399 mapread 0x001dc8 thru 0x003662 (0x189b bytes) 000420 1734671489.675718 fallocate 0x02d07e thru 0x03adee (0xdd71 bytes) 000421 1734671489.681104 trunc from 0x03adef to 0x03ff8f (0x51a1 bytes) 000422 1734671489.696366 MIRROR_EXTEND to 5 mirrors 000423 1734671490.046394 punch from 0x02b65f to 0x03b1b3 (0xfb55 bytes) 000425 1734671490.089455 fallocate 0x021ea5 thru 0x030774 (0xe8d0 bytes) 000426 1734671490.100986 fallocate 0x002894 thru 0x005a74 (0x31e1 bytes) 000427 1734671490.106978 mapwrite 0x010077 thru 0x01d4d9 (0xd463 bytes) 000428 1734671490.154525 fallocate 0x02e286 thru 0x03478f (0x650a bytes) 000429 1734671490.158349 punch from 0x00e72b to 0x00ec95 (0x56b bytes) 000430 1734671490.165621 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000431 1734671490.321495 read 0x001973 thru 0x00ea50 (0xd0de bytes) 000432 1734671490.337247 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000433 1734671490.456116 trunc from 0x03ff90 to 0x01f73c (0x20853 bytes) 000434 1734671490.483364 read 0x01212a thru 0x01c559 (0xa430 bytes) 000436 1734671490.490700 trunc from 0x01f73d to 0x024d3c (0x5600 bytes) 000438 1734671490.509822 trunc from 0x024d3d to 0x009abe (0x1b27e bytes) 000439 1734671490.524425 read 0x0041d8 thru 0x009abe (0x58e7 bytes) 000440 1734671490.530959 mapread 0x007286 thru 0x009abe (0x2839 bytes) 000441 1734671490.531701 trunc from 0x009abf to 0x00a7ce (0xd10 bytes) 000442 1734671490.542493 write 0x019c26 thru 0x023ac9 (0x9ea4 bytes) 000444 1734671490.544035 read 0x0118fe thru 0x018103 (0x6806 bytes) 000445 1734671490.549388 fallocate 0x007844 thru 0x00abc9 (0x3386 bytes) 000446 1734671490.556836 fallocate 0x010370 thru 0x01e343 (0xdfd4 bytes) 000447 1734671490.563395 punch from 0x0134f9 to 0x019457 (0x5f5f bytes) 000448 1734671490.572194 fallocate 0x02adae thru 0x02b471 (0x6c4 bytes) 000450 1734671490.575533 mapwrite 0x031aa1 thru 0x03e799 (0xccf9 bytes) 000453 1734671490.634598 write 0x00dca9 thru 0x0190dc (0xb434 bytes) 000454 1734671490.647477 read 0x02afb5 thru 0x0337dc (0x8828 bytes) 000455 1734671490.649903 fallocate 0x01a9da thru 0x01e0c5 (0x36ec bytes) 000457 1734671490.656522 mapwrite 0x0367a8 thru 0x03ffff (0x9858 bytes) 000458 1734671490.719913 trunc from 0x040000 to 0x01d4d7 (0x22b28 bytes) 000460 1734671490.736431 read 0x01a673 thru 0x01d4d7 (0x2e65 bytes) 000461 1734671490.736906 read 0x013f75 thru 0x016a79 (0x2b05 bytes) 000462 1734671490.737545 mapwrite 0x036292 thru 0x03ffff (0x9d6e bytes) 000463 1734671490.778358 punch from 0x01c66b to 0x020028 (0x39be bytes) 000464 1734671490.790391 read 0x038531 thru 0x03ffff (0x7acf bytes) 000467 1734671490.798255 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000469 1734671490.963878 read 0x01a3dc thru 0x01b408 (0x102d bytes) 000470 1734671490.968194 mapread 0x0275f6 thru 0x0365f6 (0xf001 bytes) 000473 1734671490.989457 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000474 1734671491.077941 mapread 0x01e9ec thru 0x02ae8a (0xc49f bytes) 000477 1734671491.110261 read 0x01e646 thru 0x0215f5 (0x2fb0 bytes) 000478 1734671491.110674 read 0x025228 thru 0x0276bd (0x2496 bytes) 000479 1734671491.111055 fallocate 0x02a61b thru 0x038ebf (0xe8a5 bytes) 000480 1734671491.136175 MIRROR_EXTEND to 2 mirrors 000482 1734671491.696375 mapread 0x00743b thru 0x008529 (0x10ef bytes) 000483 1734671491.704713 write 0x000447 thru 0x00d516 (0xd0d0 bytes) 000484 1734671491.743443 mapwrite 0x02b488 thru 0x03661d (0xb196 bytes) 000487 1734671491.798972 trunc from 0x040000 to 0x03f035 (0xfca bytes) 000488 1734671491.815789 trunc from 0x03f036 to 0x000d07 (0x3e32e bytes) 000489 1734671491.826510 punch from 0x000076 to 0x000d07 (0xc92 bytes) 000490 1734671491.832556 fallocate 0x01c008 thru 0x01ef85 (0x2f7e bytes) 000492 1734671491.848921 MIRROR_RESYNC 2 mirrors 000494 1734671492.112504 trunc from 0x000d08 to 0x010ea6 (0x1019f bytes) 000495 1734671492.145640 mapread 0x0101d3 thru 0x010ea6 (0xcd4 bytes) 000499 1734671492.154358 MIRROR_RESYNC 2 mirrors 000500 1734671492.392392 punch from 0x010c17 to 0x010ea6 (0x290 bytes) 000501 1734671492.429343 read 0x00882c thru 0x010ea6 (0x867b bytes) 000503 1734671492.438884 trunc from 0x010ea7 to 0x035dbc (0x24f16 bytes) 000504 1734671492.474396 read 0x0193e3 thru 0x020c43 (0x7861 bytes) 000506 1734671492.484013 mapwrite 0x02fcc3 thru 0x03dbe5 (0xdf23 bytes) 000507 1734671492.517288 mapread 0x021e4a thru 0x02df2b (0xc0e2 bytes) 000508 1734671492.518831 trunc from 0x03dbe6 to 0x020cd6 (0x1cf0f bytes) 000509 1734671492.532781 read 0x02009e thru 0x020cd6 (0xc39 bytes) 000511 1734671492.538806 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000512 1734671492.643147 fallocate 0x013202 thru 0x018c16 (0x5a15 bytes) 000516 1734671492.648283 read 0x015434 thru 0x020cd6 (0xb8a3 bytes) 000517 1734671492.654729 write 0x0028ab thru 0x0047b2 (0x1f08 bytes) 000518 1734671492.668777 trunc from 0x020cd7 to 0x021358 (0x682 bytes) 000519 1734671492.681150 punch from 0x015843 to 0x01a56b (0x4d29 bytes) 000522 1734671492.687343 write 0x00a6da thru 0x013e91 (0x97b8 bytes) 000525 1734671492.701557 mapwrite 0x0268a8 thru 0x030baa (0xa303 bytes) 000526 1734671492.774328 fallocate 0x030b43 thru 0x03884c (0x7d0a bytes) 000527 1734671492.779628 mapwrite 0x03317e thru 0x038ece (0x5d51 bytes) 000529 1734671492.839674 trunc from 0x038ecf to 0x03b8de (0x2a10 bytes) 000531 1734671492.858590 read 0x00506a thru 0x00eb9a (0x9b31 bytes) 000532 1734671492.859072 punch from 0x00f668 to 0x0111e9 (0x1b82 bytes) 000534 1734671492.863723 read 0x00bc54 thru 0x01ae98 (0xf245 bytes) 000535 1734671492.876557 trunc from 0x03b8df to 0x01d82d (0x1e0b1 bytes) 000537 1734671492.891123 mapwrite 0x02e7ed thru 0x03147b (0x2c8f bytes) 000538 1734671492.947688 write 0x0329d8 thru 0x03e1ff (0xb828 bytes) 000539 1734671492.949615 trunc from 0x03e200 to 0x03360b (0xabf4 bytes) 000542 1734671492.972338 read 0x0172af thru 0x019c0f (0x2961 bytes) 000543 1734671492.972718 mapread 0x012380 thru 0x0152a8 (0x2f29 bytes) 000544 1734671492.976071 read 0x0005be thru 0x0026ad (0x20f0 bytes) 000545 1734671492.977007 read 0x006eef thru 0x009ba5 (0x2cb7 bytes) 000546 1734671492.978031 fallocate 0x01c661 thru 0x028a65 (0xc405 bytes) 000547 1734671492.983946 read 0x022456 thru 0x023ac0 (0x166b bytes) 000548 1734671492.998709 fallocate 0x01657f thru 0x02525c (0xecde bytes) 000549 1734671493.003775 punch from 0x02ff82 to 0x03360b (0x368a bytes) 000551 1734671493.015570 trunc from 0x03360c to 0x00a17a (0x29491 bytes) 000552 1734671493.033460 write 0x02bc43 thru 0x02f0a7 (0x3465 bytes) 000553 1734671493.040357 MIRROR_EXTEND to 2 mirrors 000554 1734671493.471955 mapread 0x00500d thru 0x010f76 (0xbf6a bytes) 000555 1734671493.496020 read 0x01a2bd thru 0x01ff0e (0x5c52 bytes) 000556 1734671493.503231 read 0x025eb3 thru 0x02988b (0x39d9 bytes) 000557 1734671493.519809 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000558 1734671493.624657 write 0x027dcd thru 0x02d2b1 (0x54e5 bytes) 000559 1734671493.645145 mapwrite 0x02156f thru 0x029aa8 (0x853a bytes) 000560 1734671493.668188 mapwrite 0x026c75 thru 0x02e6b1 (0x7a3d bytes) 000561 1734671493.682632 write 0x0163ec thru 0x01c194 (0x5da9 bytes) 000562 1734671493.688101 mapread 0x003601 thru 0x009d59 (0x6759 bytes) 000563 1734671493.694520 trunc from 0x02f0a8 to 0x02fac0 (0xa19 bytes) 000565 1734671493.706457 trunc from 0x02fac1 to 0x03630a (0x684a bytes) 000566 1734671493.718550 MIRROR_EXTEND to 2 mirrors 000567 1734671494.136364 read 0x0080ef thru 0x00f202 (0x7114 bytes) 000568 1734671494.141667 read 0x00e910 thru 0x0183c0 (0x9ab1 bytes) 000569 1734671494.148368 punch from 0x030235 to 0x03630a (0x60d6 bytes) 000571 1734671494.183372 fallocate 0x011656 thru 0x0169af (0x535a bytes) 000572 1734671494.192764 MIRROR_RESYNC 2 mirrors 000573 1734671494.606303 trunc from 0x03630b to 0x01be2b (0x1a4df bytes) 000574 1734671494.645839 mapread 0x006eea thru 0x012073 (0xb18a bytes) 000575 1734671494.661251 punch from 0x00e766 to 0x01194f (0x31ea bytes) 000576 1734671494.664550 punch from 0x007309 to 0x0168bb (0xf5b3 bytes) 000577 1734671494.677772 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000578 1734671494.765569 fallocate 0x02ae6e thru 0x0372a0 (0xc433 bytes) 000580 1734671494.773002 mapread 0x00e1f0 thru 0x017bd3 (0x99e4 bytes) 000581 1734671494.790821 read 0x012cec thru 0x0162a2 (0x35b7 bytes) 000582 1734671494.793647 MIRROR_EXTEND to 2 mirrors 000584 1734671495.206414 punch from 0x0298da to 0x02be61 (0x2588 bytes) 000585 1734671495.252144 MIRROR_RESYNC 2 mirrors 000586 1734671495.576496 read 0x012be4 thru 0x01c497 (0x98b4 bytes) 000587 1734671495.583306 punch from 0x00e215 to 0x01bf89 (0xdd75 bytes) 000589 1734671495.612096 write 0x03ddea thru 0x03ffff (0x2216 bytes) 000590 1734671495.613949 punch from 0x0067a3 to 0x011730 (0xaf8e bytes) 000591 1734671495.622238 MIRROR_EXTEND to 3 mirrors 000593 1734671496.094149 punch from 0x008522 to 0x00e6d8 (0x61b7 bytes) 000595 1734671496.117937 fallocate 0x022e74 thru 0x023adf (0xc6c bytes) 000596 1734671496.123142 read 0x03c02e thru 0x03c250 (0x223 bytes) 000597 1734671496.126083 mapwrite 0x03c59e thru 0x03e1f8 (0x1c5b bytes) 000598 1734671496.143713 mapread 0x017465 thru 0x024e19 (0xd9b5 bytes) 000599 1734671496.159559 punch from 0x028b9e to 0x031b54 (0x8fb7 bytes) 000600 1734671496.163500 read 0x010460 thru 0x0114b6 (0x1057 bytes) 000601 1734671496.174112 read 0x01275e thru 0x01859f (0x5e42 bytes) 000602 1734671496.175087 fallocate 0x020709 thru 0x026fb1 (0x68a9 bytes) 000603 1734671496.180158 mapread 0x00351d thru 0x00364d (0x131 bytes) 000604 1734671496.181330 mapread 0x011e3c thru 0x02160d (0xf7d2 bytes) 000605 1734671496.183957 mapread 0x03d291 thru 0x03ffff (0x2d6f bytes) 000606 1734671496.185670 write 0x02c8c1 thru 0x035ce7 (0x9427 bytes) 000607 1734671496.192366 read 0x00adc6 thru 0x019403 (0xe63e bytes) 000609 1734671496.193884 fallocate 0x02a9c0 thru 0x02e970 (0x3fb1 bytes) 000611 1734671496.198714 fallocate 0x02b292 thru 0x037f9d (0xcd0c bytes) 000612 1734671496.203680 fallocate 0x038087 thru 0x03dfe4 (0x5f5e bytes) 000613 1734671496.207516 read 0x034686 thru 0x03ffff (0xb97a bytes) 000615 1734671496.208502 punch from 0x03f38b to 0x03ffff (0xc75 bytes) 000616 1734671496.214390 punch from 0x030172 to 0x039e09 (0x9c98 bytes) 000617 1734671496.231092 write 0x024a32 thru 0x02ca5f (0x802e bytes) 000618 1734671496.232971 trunc from 0x040000 to 0x00bf20 (0x340df bytes) 000620 1734671496.250208 trunc from 0x00bf21 to 0x017438 (0xb518 bytes) 000624 1734671496.264953 punch from 0x01418e to 0x017438 (0x32ab bytes) 000625 1734671496.268690 punch from 0x0065f8 to 0x00b461 (0x4e6a bytes) 000626 1734671496.275441 MIRROR_EXTEND to 4 mirrors 000627 1734671496.574834 fallocate 0x030181 thru 0x03d238 (0xd0b8 bytes) 000628 1734671496.608868 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000631 1734671496.734492 fallocate 0x00636e thru 0x00cc7a (0x690d bytes) 000632 1734671496.739149 trunc from 0x017439 to 0x03e4b2 (0x2707a bytes) 000633 1734671496.772243 MIRROR_EXTEND to 4 mirrors 000634 1734671497.080955 mapread 0x01795f thru 0x01b8e5 (0x3f87 bytes) 000636 1734671497.089766 mapread 0x03831c thru 0x03bf4b (0x3c30 bytes) 000637 1734671497.098254 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000638 1734671497.319329 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000639 1734671497.479570 mapwrite 0x0012af thru 0x004715 (0x3467 bytes) 000640 1734671497.571339 mapwrite 0x0034de thru 0x00d381 (0x9ea4 bytes) 000641 1734671497.595620 trunc from 0x03e4b3 to 0x03f890 (0x13de bytes) 000642 1734671497.623376 trunc from 0x03f891 to 0x02bc34 (0x13c5c bytes) 000645 1734671497.642361 mapread 0x00a855 thru 0x00dcc0 (0x346c bytes) 000646 1734671497.643856 mapwrite 0x02ed18 thru 0x031252 (0x253b bytes) 000647 1734671497.688222 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000648 1734671497.767166 trunc from 0x031253 to 0x013f4c (0x1d306 bytes) 000649 1734671497.791078 mapwrite 0x027f1c thru 0x032972 (0xaa57 bytes) 000650 1734671497.853786 fallocate 0x00cadf thru 0x013406 (0x6928 bytes) 000651 1734671497.861007 mapwrite 0x021ebf thru 0x028444 (0x6586 bytes) 000652 1734671497.893269 fallocate 0x006478 thru 0x00ef40 (0x8ac9 bytes) 000653 1734671497.896884 read 0x018eab thru 0x025ada (0xcc30 bytes) 000654 1734671497.906667 read 0x009c74 thru 0x00b1af (0x153c bytes) 000655 1734671497.926293 MIRROR_EXTEND to 2 mirrors 000656 1734671498.279703 trunc from 0x032973 to 0x02f978 (0x2ffa bytes) 000657 1734671498.312064 MIRROR_EXTEND to 3 mirrors 000658 1734671498.670443 trunc from 0x02f979 to 0x01ae94 (0x14ae4 bytes) 000660 1734671498.712363 fallocate 0x019912 thru 0x01c50c (0x2bfb bytes) 000662 1734671498.716706 fallocate 0x01af3d thru 0x02a900 (0xf9c4 bytes) 000663 1734671498.721457 trunc from 0x01c50d to 0x0236e2 (0x71d6 bytes) 000664 1734671498.736493 mapread 0x021e13 thru 0x0236e2 (0x18d0 bytes) 000665 1734671498.749721 trunc from 0x0236e3 to 0x0206f6 (0x2fec bytes) 000666 1734671498.762385 mapwrite 0x020f47 thru 0x02e563 (0xd61d bytes) 000667 1734671498.827554 punch from 0x000311 to 0x00063b (0x32b bytes) 000668 1734671498.834175 read 0x0204b4 thru 0x02e563 (0xe0b0 bytes) 000669 1734671498.834996 mapread 0x001ec6 thru 0x01056d (0xe6a8 bytes) 000670 1734671498.846628 mapwrite 0x009cf9 thru 0x00ded9 (0x41e1 bytes) 000671 1734671498.859174 read 0x00e704 thru 0x01b778 (0xd075 bytes) 000673 1734671498.867132 read 0x01d955 thru 0x024679 (0x6d25 bytes) 000674 1734671498.867632 trunc from 0x02e564 to 0x0149b4 (0x19baf bytes) 000675 1734671498.881251 write 0x00ab2a thru 0x015caa (0xb181 bytes) 000676 1734671498.883807 write 0x012f28 thru 0x01b103 (0x81dc bytes) 000677 1734671498.886314 punch from 0x005046 to 0x006d1d (0x1cd8 bytes) 000678 1734671498.896623 trunc from 0x01b104 to 0x02da50 (0x1294d bytes) 000679 1734671498.906991 punch from 0x001b96 to 0x00ffff (0xe46a bytes) 000680 1734671498.915418 read 0x003bbd thru 0x00fb88 (0xbfcc bytes) 000682 1734671498.926134 MIRROR_RESYNC 3 mirrors 000683 1734671499.470250 trunc from 0x02da51 to 0x0245fc (0x9454 bytes) 000684 1734671499.524677 fallocate 0x012c72 thru 0x021d8b (0xf11a bytes) 000685 1734671499.533047 write 0x03b64b thru 0x03ffff (0x49b5 bytes) 000687 1734671499.534961 mapwrite 0x03fe7d thru 0x03ffff (0x183 bytes) 000689 1734671499.557147 read 0x020993 thru 0x029c69 (0x92d7 bytes) 000690 1734671499.564671 write 0x0090fe thru 0x0113af (0x82b2 bytes) 000691 1734671499.574359 fallocate 0x016954 thru 0x017e22 (0x14cf bytes) 000692 1734671499.580586 mapwrite 0x00df8b thru 0x010555 (0x25cb bytes) 000693 1734671499.597769 mapwrite 0x0075dc thru 0x00e016 (0x6a3b bytes) 000694 1734671499.624887 trunc from 0x040000 to 0x035d3a (0xa2c5 bytes) 000695 1734671499.638941 fallocate 0x01c857 thru 0x02178f (0x4f39 bytes) 000697 1734671499.643627 punch from 0x0215d1 to 0x02a1e8 (0x8c18 bytes) 000700 1734671499.649113 read 0x02db03 thru 0x032eb5 (0x53b3 bytes) 000701 1734671499.655473 mapwrite 0x026cd7 thru 0x0279f5 (0xd1f bytes) 000702 1734671499.666445 read 0x00f2ed thru 0x0102aa (0xfbe bytes) 000703 1734671499.666935 punch from 0x033315 to 0x035d3a (0x2a26 bytes) 000704 1734671499.672665 read 0x002509 thru 0x006641 (0x4139 bytes) 000705 1734671499.673753 mapread 0x001f62 thru 0x002648 (0x6e7 bytes) 000706 1734671499.677309 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000707 1734671499.825302 fallocate 0x0214ae thru 0x029406 (0x7f59 bytes) 000708 1734671499.830123 trunc from 0x035d3b to 0x02cec3 (0x8e77 bytes) 000709 1734671499.844442 read 0x019135 thru 0x022018 (0x8ee4 bytes) 000712 1734671499.852357 fallocate 0x02a86a thru 0x03116b (0x6902 bytes) 000713 1734671499.858955 write 0x01c5e3 thru 0x01d1b4 (0xbd2 bytes) 000714 1734671499.859622 punch from 0x02e445 to 0x03116b (0x2d27 bytes) 000715 1734671499.864711 write 0x0021b9 thru 0x005d41 (0x3b89 bytes) 000716 1734671499.872041 fallocate 0x0207ce thru 0x02e243 (0xda76 bytes) 000717 1734671499.875618 punch from 0x020b2c to 0x02e522 (0xd9f7 bytes) 000718 1734671499.878947 trunc from 0x03116c to 0x00780f (0x2995c bytes) 000719 1734671499.890426 fallocate 0x0046e3 thru 0x00cd92 (0x86b0 bytes) 000720 1734671499.894359 punch from 0x004ab6 to 0x00780f (0x2d5a bytes) 000721 1734671499.910081 punch from 0x000e5a to 0x00780f (0x69b6 bytes) 000722 1734671499.916894 read 0x003ae8 thru 0x00780f (0x3d28 bytes) 000723 1734671499.933611 MIRROR_RESYNC 2 mirrors 000724 1734671500.307946 fallocate 0x033e19 thru 0x03b786 (0x796e bytes) 000725 1734671500.360164 MIRROR_EXTEND to 3 mirrors 000726 1734671500.747487 mapwrite 0x027ee6 thru 0x02d441 (0x555c bytes) 000727 1734671500.809011 fallocate 0x005837 thru 0x00fc61 (0xa42b bytes) 000728 1734671500.828150 MIRROR_EXTEND to 4 mirrors 000729 1734671501.272681 fallocate 0x029cb2 thru 0x035908 (0xbc57 bytes) 000730 1734671501.312953 MIRROR_RESYNC 4 mirrors 000731 1734671501.894144 read 0x021b2f thru 0x02d441 (0xb913 bytes) 000732 1734671501.902076 read 0x027fb0 thru 0x02d441 (0x5492 bytes) 000733 1734671501.902392 fallocate 0x03469f thru 0x037ce6 (0x3648 bytes) 000734 1734671501.939475 read 0x022311 thru 0x02d441 (0xb131 bytes) 000735 1734671501.946376 punch from 0x02b195 to 0x02d441 (0x22ad bytes) 000736 1734671501.949620 mapwrite 0x011de5 thru 0x014aad (0x2cc9 bytes) 000737 1734671501.961378 fallocate 0x02adee thru 0x032ce8 (0x7efb bytes) 000739 1734671501.966305 punch from 0x00e242 to 0x01a816 (0xc5d5 bytes) 000740 1734671501.969546 mapread 0x0007b8 thru 0x008b25 (0x836e bytes) 000743 1734671501.971957 write 0x030868 thru 0x03b6f1 (0xae8a bytes) 000746 1734671501.975711 trunc from 0x03b6f2 to 0x0184fa (0x231f7 bytes) 000747 1734671501.996840 punch from 0x016c8a to 0x0184fa (0x1871 bytes) 000748 1734671502.004221 mapwrite 0x01c22a thru 0x01cf21 (0xcf8 bytes) 000749 1734671502.041321 read 0x00978e thru 0x011bdd (0x8450 bytes) 000750 1734671502.050198 trunc from 0x01cf22 to 0x007d65 (0x151bc bytes) 000751 1734671502.063947 mapwrite 0x029f76 thru 0x039d98 (0xfe23 bytes) 000752 1734671502.103949 trunc from 0x039d99 to 0x01c939 (0x1d45f bytes) 000755 1734671502.129292 MIRROR_EXTEND to 5 mirrors 000756 1734671502.494008 fallocate 0x03ca4c thru 0x03ffff (0x35b4 bytes) 000757 1734671502.519247 mapwrite 0x000080 thru 0x00c49c (0xc41d bytes) 000759 1734671502.564838 write 0x037195 thru 0x037fd8 (0xe44 bytes) 000760 1734671502.587813 MIRROR_RESYNC 5 mirrors 000761 1734671503.283936 mapread 0x036d10 thru 0x03eab2 (0x7da3 bytes) 000763 1734671503.298227 punch from 0x01d457 to 0x02c618 (0xf1c2 bytes) 000764 1734671503.328596 mapread 0x002ee2 thru 0x00fa2f (0xcb4e bytes) 000765 1734671503.340037 punch from 0x035b82 to 0x03cecb (0x734a bytes) 000766 1734671503.343462 fallocate 0x00ee43 thru 0x01a7bc (0xb97a bytes) 000768 1734671503.349718 MIRROR_RESYNC 5 mirrors 000769 1734671503.853164 write 0x003505 thru 0x0078c7 (0x43c3 bytes) 000771 1734671503.885573 read 0x024fa3 thru 0x02be58 (0x6eb6 bytes) 000772 1734671503.892321 punch from 0x0015d5 to 0x003c24 (0x2650 bytes) 000773 1734671503.907632 MIRROR_EXTEND to 6 mirrors 000774 1734671504.319448 fallocate 0x026c6c thru 0x027930 (0xcc5 bytes) 000775 1734671504.362050 write 0x0216ac thru 0x028445 (0x6d9a bytes) 000777 1734671504.372683 mapread 0x02fa79 thru 0x03927f (0x9807 bytes) 000778 1734671504.384981 punch from 0x0178a5 to 0x01ffa7 (0x8703 bytes) 000780 1734671504.390094 mapread 0x026ae2 thru 0x02c8ce (0x5ded bytes) 000781 1734671504.392522 punch from 0x03f02b to 0x03ffff (0xfd5 bytes) 000782 1734671504.404647 MIRROR_RESYNC 6 mirrors 000783 1734671505.576763 read 0x03b720 thru 0x03ffff (0x48e0 bytes) 000784 1734671505.585541 trunc from 0x040000 to 0x01ca8f (0x23570 bytes) 000785 1734671505.627485 mapwrite 0x011e20 thru 0x01a998 (0x8b79 bytes) 000786 1734671505.664427 write 0x029b4b thru 0x033384 (0x983a bytes) 000787 1734671505.665448 read 0x0286a1 thru 0x02d7b2 (0x5112 bytes) 000788 1734671505.675036 read 0x00a8c6 thru 0x010b6d (0x62a8 bytes) 000789 1734671505.675470 read 0x002c5b thru 0x00c18e (0x9534 bytes) 000790 1734671505.675890 fallocate 0x020503 thru 0x027558 (0x7056 bytes) 000791 1734671505.683541 trunc from 0x033385 to 0x02929b (0xa0e9 bytes) 000792 1734671505.700942 write 0x027802 thru 0x03046b (0x8c6a bytes) 000793 1734671505.702063 punch from 0x026555 to 0x02dae0 (0x758c bytes) 000794 1734671505.717775 read 0x015906 thru 0x024e45 (0xf540 bytes) 000795 1734671505.718307 punch from 0x01653c to 0x01a242 (0x3d07 bytes) 000796 1734671505.721706 trunc from 0x03046c to 0x007bc4 (0x288a7 bytes) 000797 1734671505.751932 MIRROR_RESYNC 6 mirrors 000799 1734671506.351362 mapread 0x0030dc thru 0x007349 (0x426e bytes) 000800 1734671506.363317 fallocate 0x024895 thru 0x033e19 (0xf585 bytes) 000801 1734671506.394484 MIRROR_RESYNC 6 mirrors 000802 1734671507.206184 mapwrite 0x0339f9 thru 0x03ffff (0xc607 bytes) 000803 1734671507.319235 mapwrite 0x01b303 thru 0x01c506 (0x1204 bytes) 000804 1734671507.350309 mapread 0x00cb90 thru 0x012e33 (0x62a4 bytes) 000806 1734671507.368961 mapread 0x02909a thru 0x037090 (0xdff7 bytes) 000807 1734671507.387917 read 0x016640 thru 0x01f322 (0x8ce3 bytes) 000808 1734671507.389786 fallocate 0x016c8a thru 0x0188c1 (0x1c38 bytes) 000809 1734671507.401252 mapwrite 0x00c4d2 thru 0x010f2a (0x4a59 bytes) 000810 1734671507.418985 mapwrite 0x005692 thru 0x009531 (0x3ea0 bytes) 000813 1734671507.494353 MIRROR_RESYNC 6 mirrors 000814 1734671508.459301 trunc from 0x040000 to 0x010165 (0x2fe9a bytes) 000815 1734671508.490566 write 0x00ee44 thru 0x012a9b (0x3c58 bytes) 000816 1734671508.499724 MIRROR_EXTEND to 7 mirrors 000817 1734671508.893854 read 0x00cdb8 thru 0x012a9b (0x5ce4 bytes) 000819 1734671508.925386 trunc from 0x012a9c to 0x0350ad (0x22612 bytes) 000822 1734671508.985269 punch from 0x01c701 to 0x024afa (0x83fa bytes) 000823 1734671508.991866 read 0x01e075 thru 0x0252ca (0x7256 bytes) 000824 1734671509.010120 read 0x00f6c7 thru 0x014d03 (0x563d bytes) 000825 1734671509.010549 read 0x013660 thru 0x020aa3 (0xd444 bytes) 000826 1734671509.013690 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000827 1734671509.176686 write 0x030713 thru 0x034a11 (0x42ff bytes) 000828 1734671509.187668 fallocate 0x0063f1 thru 0x012aae (0xc6be bytes) 000831 1734671509.195667 fallocate 0x01e9f0 thru 0x02c4bb (0xdacc bytes) 000832 1734671509.199805 mapwrite 0x024f19 thru 0x030967 (0xba4f bytes) 000833 1734671509.238995 mapwrite 0x03e075 thru 0x03ffff (0x1f8b bytes) 000834 1734671509.299012 read 0x013d19 thru 0x019010 (0x52f8 bytes) 000835 1734671509.311156 write 0x0113b6 thru 0x01254e (0x1199 bytes) 000836 1734671509.320913 fallocate 0x019cf7 thru 0x01ed99 (0x50a3 bytes) 000837 1734671509.324987 fallocate 0x030c7a thru 0x03f233 (0xe5ba bytes) 000838 1734671509.334511 trunc from 0x040000 to 0x007d7c (0x38283 bytes) 000839 1734671509.355465 punch from 0x000c6b to 0x0031e5 (0x257b bytes) 000843 1734671509.369142 mapread 0x0056f1 thru 0x005ecc (0x7dc bytes) 000845 1734671509.386999 punch from 0x007b8a to 0x007d7c (0x1f3 bytes) 000846 1734671509.397056 write 0x00d64a thru 0x017083 (0x9a3a bytes) 000847 1734671509.404697 mapwrite 0x03cd14 thru 0x03ce38 (0x125 bytes) 000848 1734671509.452170 read 0x02c349 thru 0x034879 (0x8531 bytes) 000849 1734671509.467625 fallocate 0x03dcad thru 0x03ffff (0x2353 bytes) 000850 1734671509.473851 trunc from 0x040000 to 0x012e73 (0x2d18c bytes) 000853 1734671509.490350 mapread 0x00a96b thru 0x012e73 (0x8509 bytes) 000854 1734671509.492005 trunc from 0x012e74 to 0x011baa (0x12c9 bytes) 000855 1734671509.503463 mapread 0x00b966 thru 0x011baa (0x6245 bytes) 000857 1734671509.505080 read 0x00454d thru 0x0106b2 (0xc166 bytes) 000858 1734671509.505517 fallocate 0x021121 thru 0x028eac (0x7d8c bytes) 000859 1734671509.512110 write 0x03a5e3 thru 0x03ffff (0x5a1d bytes) 000860 1734671509.513496 mapwrite 0x035cff thru 0x03ffff (0xa301 bytes) 000862 1734671509.544054 write 0x02a7d4 thru 0x02dece (0x36fb bytes) 000863 1734671509.550767 mapwrite 0x01fd0b thru 0x02a46b (0xa761 bytes) 000864 1734671509.571117 read 0x03f221 thru 0x03ffff (0xddf bytes) 000865 1734671509.571542 mapread 0x019926 thru 0x020844 (0x6f1f bytes) 000867 1734671509.572455 trunc from 0x040000 to 0x03f317 (0xce8 bytes) 000868 1734671509.586631 punch from 0x01a4c9 to 0x029dde (0xf916 bytes) 000869 1734671509.590964 read 0x006b71 thru 0x00c40d (0x589d bytes) 000870 1734671509.591416 write 0x00dc96 thru 0x01843a (0xa7a5 bytes) 000871 1734671509.593323 write 0x0141b5 thru 0x01ad5b (0x6ba7 bytes) 000872 1734671509.598707 mapread 0x011b9b thru 0x017fd5 (0x643b bytes) 000873 1734671509.599823 fallocate 0x027078 thru 0x02a05e (0x2fe7 bytes) 000874 1734671509.604221 trunc from 0x03f318 to 0x024ee1 (0x1a436 bytes) 000875 1734671509.617500 trunc from 0x024ee2 to 0x02a8e7 (0x5a06 bytes) 000876 1734671509.630311 write 0x035ff9 thru 0x03b3a6 (0x53ae bytes) 000877 1734671509.631219 fallocate 0x011c20 thru 0x02141a (0xf7fb bytes) 000879 1734671509.636610 read 0x01996a thru 0x01ca60 (0x30f7 bytes) 000880 1734671509.637024 mapwrite 0x00e10b thru 0x01dedf (0xfdd5 bytes) 000881 1734671509.655381 mapwrite 0x03ec3c thru 0x03ffff (0x13c4 bytes) 000882 1734671509.686250 mapwrite 0x01a574 thru 0x01ea9f (0x452c bytes) 000883 1734671509.700389 mapread 0x031a7e thru 0x03ee7f (0xd402 bytes) 000885 1734671509.701387 read 0x02d800 thru 0x02fb7b (0x237c bytes) 000886 1734671509.701790 trunc from 0x040000 to 0x01c6b2 (0x2394d bytes) 000887 1734671509.717534 mapwrite 0x03466d thru 0x034e0a (0x79e bytes) 000888 1734671509.748616 mapwrite 0x039db9 thru 0x03bf23 (0x216b bytes) 000889 1734671509.786932 mapread 0x014aa5 thru 0x01b797 (0x6cf3 bytes) 000890 1734671509.794478 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000891 1734671509.956157 fallocate 0x028809 thru 0x02c1f7 (0x39ef bytes) 000892 1734671509.961905 punch from 0x023b61 to 0x02561f (0x1abf bytes) 000893 1734671509.965354 read 0x034134 thru 0x036aa3 (0x2970 bytes) 000894 1734671509.970077 write 0x01bc01 thru 0x029d63 (0xe163 bytes) 000895 1734671509.985941 punch from 0x0399df to 0x03bf23 (0x2545 bytes) 000896 1734671509.989797 trunc from 0x03bf24 to 0x03638b (0x5b98 bytes) 000897 1734671510.004417 mapread 0x019c50 thru 0x026761 (0xcb12 bytes) 000899 1734671510.009616 read 0x020cc1 thru 0x02be97 (0xb1d7 bytes) 000900 1734671510.020012 mapwrite 0x004f87 thru 0x0059e5 (0xa5f bytes) 000901 1734671510.036276 punch from 0x00ee9b to 0x01de80 (0xefe6 bytes) 000902 1734671510.040082 punch from 0x02a6d8 to 0x02fde0 (0x5709 bytes) 000903 1734671510.046621 fallocate 0x01d527 thru 0x021bd1 (0x46ab bytes) 000905 1734671510.051995 mapread 0x018d55 thru 0x026d07 (0xdfb3 bytes) 000907 1734671510.075508 fallocate 0x02b0fa thru 0x03a3e4 (0xf2eb bytes) 000908 1734671510.080478 read 0x02b215 thru 0x030515 (0x5301 bytes) 000909 1734671510.088243 mapwrite 0x016e37 thru 0x01f81f (0x89e9 bytes) 000910 1734671510.104288 punch from 0x009892 to 0x015e8b (0xc5fa bytes) 000911 1734671510.110311 read 0x0359bc thru 0x03a3e4 (0x4a29 bytes) 000912 1734671510.110781 punch from 0x0318f7 to 0x03a3e4 (0x8aee bytes) 000914 1734671510.116820 punch from 0x00453d to 0x0122aa (0xdd6e bytes) 000917 1734671510.121721 mapread 0x00dd49 thru 0x018d64 (0xb01c bytes) 000918 1734671510.132808 punch from 0x034776 to 0x03a3e4 (0x5c6f bytes) 000919 1734671510.144656 MIRROR_RESYNC 5 mirrors 000920 1734671511.341528 fallocate 0x016a26 thru 0x01ac2e (0x4209 bytes) 000921 1734671511.374210 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000923 1734671511.522173 read 0x01e264 thru 0x02c0d7 (0xde74 bytes) 000924 1734671511.530301 mapread 0x037385 thru 0x03a3e4 (0x3060 bytes) 000925 1734671511.551122 write 0x00b508 thru 0x00d82f (0x2328 bytes) 000927 1734671511.560196 write 0x02dde3 thru 0x037c4c (0x9e6a bytes) 000928 1734671511.566889 read 0x02a71d thru 0x02e62c (0x3f10 bytes) 000929 1734671511.567371 trunc from 0x03a3e5 to 0x0192d4 (0x21110 bytes) 000930 1734671511.582029 punch from 0x0036ff to 0x00ca8e (0x9390 bytes) 000931 1734671511.595382 mapwrite 0x03cab9 thru 0x03ffff (0x3547 bytes) 000932 1734671511.623636 MIRROR_EXTEND to 5 mirrors 000933 1734671512.079823 write 0x03ceac thru 0x03ffff (0x3154 bytes) 000934 1734671512.122834 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000935 1734671512.267478 write 0x01e5df thru 0x026312 (0x7d34 bytes) 000936 1734671512.279672 fallocate 0x0195d4 thru 0x01b381 (0x1dae bytes) 000937 1734671512.283365 read 0x01b335 thru 0x027836 (0xc502 bytes) 000938 1734671512.297481 write 0x00dca8 thru 0x00e3ec (0x745 bytes) 000939 1734671512.309473 mapwrite 0x006277 thru 0x008cf5 (0x2a7f bytes) 000941 1734671512.324573 fallocate 0x014318 thru 0x01a1f7 (0x5ee0 bytes) 000942 1734671512.327995 trunc from 0x040000 to 0x02a110 (0x15eef bytes) 000943 1734671512.351301 fallocate 0x03b832 thru 0x03ffff (0x47ce bytes) 000944 1734671512.358075 mapwrite 0x02781e thru 0x02c2cc (0x4aaf bytes) 000945 1734671512.391233 read 0x015efe thru 0x023e3f (0xdf42 bytes) 000946 1734671512.396463 trunc from 0x02c2cd to 0x00007d (0x2c24f bytes) 000947 1734671512.423113 mapread 0x000045 thru 0x00007d (0x39 bytes) 000948 1734671512.424194 trunc from 0x00007e to 0x034281 (0x34204 bytes) 000950 1734671512.441134 read 0x008190 thru 0x0162e3 (0xe154 bytes) 000951 1734671512.454147 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000953 1734671512.610675 mapread 0x00b611 thru 0x00e6b5 (0x30a5 bytes) 000954 1734671512.635810 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000955 1734671512.792248 punch from 0x023e11 to 0x0301d4 (0xc3c4 bytes) 000956 1734671512.797530 fallocate 0x01c512 thru 0x02922f (0xcd1e bytes) 000957 1734671512.804829 mapwrite 0x01f801 thru 0x01fdca (0x5ca bytes) 000958 1734671512.838148 write 0x024b79 thru 0x03054f (0xb9d7 bytes) 000959 1734671512.864355 MIRROR_RESYNC 2 mirrors 000960 1734671513.304249 trunc from 0x034282 to 0x008b68 (0x2b719 bytes) 000962 1734671513.344291 mapread 0x00883f thru 0x008b68 (0x32a bytes) 000963 1734671513.352703 write 0x02bed4 thru 0x03647b (0xa5a8 bytes) 000964 1734671513.358385 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x31e:0x0]: Invalid argument (22) 000965 1734671513.478059 trunc from 0x03647c to 0x037c42 (0x17c7 bytes) 000966 1734671513.510010 mapwrite 0x0150a3 thru 0x015b7d (0xadb bytes) 000968 1734671513.541141 fallocate 0x029012 thru 0x02d086 (0x4075 bytes) 000969 1734671513.545595 punch from 0x00ca92 to 0x015ece (0x943d bytes) 000970 1734671513.550972 punch from 0x005459 to 0x0149ab (0xf553 bytes) 000971 1734671513.554293 mapwrite 0x0371a9 thru 0x03ffff (0x8e57 bytes) 000972 1734671513.610672 mapwrite 0x0216f9 thru 0x0246f2 (0x2ffa bytes) 000975 1734671513.650358 mapwrite 0x03c316 thru 0x03ffff (0x3cea bytes) 000976 1734671513.663355 write 0x0374b0 thru 0x03a3aa (0x2efb bytes) 000977 1734671513.665384 mapwrite 0x00f8e3 thru 0x01ee2a (0xf548 bytes) 000978 1734671513.692794 mapread 0x00541a thru 0x008090 (0x2c77 bytes) 000979 1734671513.699537 trunc from 0x040000 to 0x0324ca (0xdb35 bytes) 000981 1734671513.717180 mapread 0x027b4a thru 0x02eeba (0x7371 bytes) 000982 1734671513.724438 fallocate 0x03fa84 thru 0x03ffff (0x57c bytes) 000983 1734671513.729372 write 0x035d66 thru 0x03c3e8 (0x6683 bytes) 000984 1734671513.735625 mapread 0x028a24 thru 0x033719 (0xacf6 bytes) 000985 1734671513.737218 mapread 0x01652b thru 0x01d513 (0x6fe9 bytes) 000986 1734671513.738186 write 0x031ebc thru 0x039e06 (0x7f4b bytes) 000987 1734671513.739922 fallocate 0x033baa thru 0x03ee7d (0xb2d4 bytes) 000988 1734671513.743447 punch from 0x037b64 to 0x03ffff (0x849c bytes) 000989 1734671513.753993 punch from 0x0174fe to 0x01edb3 (0x78b6 bytes) 000990 1734671513.759824 write 0x006f4e thru 0x0130ce (0xc181 bytes) 000991 1734671513.761434 punch from 0x02b553 to 0x03406f (0x8b1d bytes) 000992 1734671513.767937 trunc from 0x040000 to 0x00e0b6 (0x31f49 bytes) 000993 1734671513.784144 write 0x03453c thru 0x03ed15 (0xa7da bytes) 000994 1734671513.785308 mapread 0x0289a3 thru 0x02c417 (0x3a75 bytes) 000995 1734671513.797914 mapwrite 0x002310 thru 0x01022e (0xdf1f bytes) 000997 1734671513.822419 mapread 0x03c5ef thru 0x03ed15 (0x2727 bytes) 001000 1734671513.823779 mapwrite 0x002e29 thru 0x00f8cb (0xcaa3 bytes) All operations completed A-OK! PASS 70a (45s) == sanity-flr test 200a: stress test ===================== 00:11:56 (1734671516) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8253280 .. Extending file size to 1507904 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8401984 .. Extending file size to 3245632 .. Extending file size to 7818240 .. Extending file size to 8710112 .. Extending file size to 9124512 .. Extending file size to 8744480 .. Extending file size to 9113728 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9204576 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 30846 30847 30848 30851 30852 done 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) 192.168.204.130@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg430-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (69s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 00:13:05 (1734671585) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4894496 .. Extending file size to 2922112 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4116128 .. Extending file size to 8071168 .. Extending file size to 6088256 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8222752 .. Extending file size to 8664800 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8676384 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8820672 .. Extending file size to 9025120 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 32674 32675 32676 32679 32680 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) 192.168.204.130@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg430-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (68s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 00:14:14 (1734671654) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2320 1285368 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2340 1285348 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15584 3591436 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21212 3585808 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 36796 7177244 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2320 1285368 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2340 1285348 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15584 3591436 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 21212 3585808 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 36796 7177244 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.051286 s, 40.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.120684 s, 17.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 00:14:23 (1734671663) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0581231 s, 36.1 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 00:14:27 (1734671667) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8b:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 00:14:30 (1734671670) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.057768 s, 36.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x390:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x390:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x391:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 00:14:34 (1734671674) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0592882 s, 35.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 00:14:38 (1734671678) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0566024 s, 37.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } PASS 204c (26s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 00:15:04 (1734671704) Waiting for MDT destroys to complete oleg430-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg430-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg430-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.028904 s, 36.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x391:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x393:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x392:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x391:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x393:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x392:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x392:0x0] } PASS 204d (36s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 00:15:40 (1734671740) Waiting for MDT destroys to complete oleg430-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 oleg430-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg430-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0641125 s, 32.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 00:16:15 (1734671775) Waiting for MDT destroys to complete oleg430-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 oleg430-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg430-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg430-server: Pool lustre.test_204f created Adding targets to pool oleg430-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg430-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0596549 s, 35.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x394:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x393:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x394:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x393:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg430-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg430-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg430-server: Pool lustre.test_204f destroyed PASS 204f (51s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 00:17:06 (1734671826) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 00:17:10 (1734671830) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 00:17:13 (1734671833) oleg430-server: Pool lustre.test_206 created Adding targets to pool oleg430-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg430-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg430-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg430-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg430-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (19s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 00:17:32 (1734671852) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0387478 s, 27.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0190252 s, 55.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x676e3958 Mirror 3: 0x676e3958 PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 00:17:37 (1734671857) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 00:17:39 (1734671859) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 00:17:41 (1734671861) Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0418794 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2540000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x14a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x14a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x14a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1950000 Read this (4 bytes): BBBB 192.168.204.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg430-client.virtnet /mnt/lustre2 (opts:) PASS 209a (20s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 00:18:01 (1734671881) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0109378 s, 749 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 00:18:05 (1734671885) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0345073 s, 30.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xba:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-20 00:18:06.000000000 -0500 Modify: 2024-12-20 00:18:06.000000000 -0500 Change: 2024-12-20 00:18:06.000000000 -0500 Birth: - PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 00:18:09 (1734671889) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0211698 s, 49.5 MB/s oleg430-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg430-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg430-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 oleg430-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 oleg430-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg430-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg430-server: O/280000401/d27/bb: File not found by ext2_lookup PASS 210b (17s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 00:18:26 (1734671906) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.132925 s, 308 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0100691 s, 407 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 20 00:18 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x84:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 20 00:18 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test complete, duration 1098 sec =========== 00:18:30 (1734671910) === sanity-flr: start cleanup 00:18:31 (1734671911) === === sanity-flr: finish cleanup 00:18:35 (1734671915) ===