-----============= acceptance-small: sanity-flr ============----- Tue Oct 1 02:26:45 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c 49a === sanity-flr: start setup 02:26:54 (1727764014) === oleg303-client.virtnet: executing check_config_client /mnt/lustre oleg303-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg303-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b3f74800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b3f74800.idle_timeout=debug disable quota as required oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 02:27:06 (1727764026) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6980] == sanity-flr test 0a: lfs mirror create with -N option == 02:27:06 (1727764026) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:27:14 (1727764034) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:27:16 (1727764036) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:27:18 (1727764038) PASS 0d (11s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:27:29 (1727764049) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:27:31 (1727764051) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 02:27:32 (1727764052) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b3f74800.stats=clear osc.lustre-OST0001-osc-ffff8800b3f74800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:27:36 (1727764056) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:27:42 (1727764062) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:27:46 (1727764066) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:27:47 (1727764067) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 02:27:51 (1727764071) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:27:55 (1727764075) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:27:56 (1727764076) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:27:59 (1727764079) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:28:04 (1727764084) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:28:07 (1727764087) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 1.3976 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106646 s, 9.8 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:28:13 (1727764093) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 0.215602 s, 87.5 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:28:17 (1727764097) Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3f74800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3f74800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b3f74800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3f74800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg303-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (27s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:28:44 (1727764124) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.354872 s, 32.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3f74800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3f74800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (17s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:29:01 (1727764141) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3f74800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3f74800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b3f74800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3f74800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg303-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (36s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:29:38 (1727764178) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.34503 s, 39.0 MB/s -rw-r--r-- 1 root root 52428800 Oct 1 02:29 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.452s user 0m0.002s sys 0m0.134s Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3f74800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3f74800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.382s user 0m0.000s sys 0m0.146s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b3f74800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3f74800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.483s user 0m0.000s sys 0m0.137s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg303-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b3f74800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3f74800.ost_server_uuid in FULL state after 0 sec PASS 33b (37s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:30:14 (1727764214) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:30:16 (1727764216) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:30:17 (1727764217) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 02:30:18 (1727764218) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 02:30:21 (1727764221) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.230549 s, 54.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:30:27 (1727764227) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163177 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0236617 s, 44.3 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:30:30 (1727764230) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:30:35 (1727764235) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000893348 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000859001 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0033728 s, 304 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00158844 s, 645 kB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 02:30:39 (1727764239) create test files with size 15748373 .. bcc8c52b8ff65fa23f3a7bf294d110ac - 893bc5c0a1f30bceba7d905e1984d00e - f7106b2229324b0b1c82e35d6ff3e63b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre PASS 37 (13s) == sanity-flr test 38: resync ============================ 02:30:52 (1727764252) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:30:59 (1727764259) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:31:02 (1727764262) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0734893 s, 14.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017334 s, 60.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 02:31:06 (1727764266) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0754767 s, 27.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.173741 s, 24.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.107652 s, 19.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 02:31:11 (1727764271) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 02:31:13 (1727764273) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:31:15 (1727764275) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215168 s, 48.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 02:31:19 (1727764279) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:31:20 (1727764280) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x13:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:31:24 (1727764284) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0985076 s, 106 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (4s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:31:28 (1727764288) llite.lustre-ffff88012d47f000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.117804 s, 89.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 36878 flags: 1 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 36878 flags: 1 llite.lustre-ffff8800b66ec000.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:31:34 (1727764294) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:31:45 (1727764305) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (9s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:31:54 (1727764314) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 02:31:56 (1727764316) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.113049 s, 27.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.073004 s, 43.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 78918ef45ec4a178d1639d837b31ddb1 - mirror 1 checksum: 4b1f32eee0a2df23d02f47c29da26b36 - mirror 2 checksum: 78918ef45ec4a178d1639d837b31ddb1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Oct 1 02:31:56 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Oct 1 02:31:56 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 78918ef45ec4a178d1639d837b31ddb1 - mirror 1 checksum: 4b1f32eee0a2df23d02f47c29da26b36 - mirror 2 checksum: 4b1f32eee0a2df23d02f47c29da26b36 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Oct 1 02:31:56 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Oct 1 02:31:56 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:32:00 (1727764320) Failing mds1 on oleg303-server Stopping /mnt/lustre-mds1 (opts:) on oleg303-server 02:32:02 (1727764322) shut down Failover mds1 to oleg303-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 02:32:18 (1727764338) targets are mounted 02:32:18 (1727764338) facet_failover done oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0149371 s, 70.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 50A (25s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:32:25 (1727764345) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:32:27 (1727764347) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:32:29 (1727764349) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00384263 s, 4.3 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:32:32 (1727764352) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:32:35 (1727764355) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:32:37 (1727764357) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (11s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:32:48 (1727764368) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:33:08 (1727764388) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:33:26 (1727764406) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:33:45 (1727764425) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 02:33:49 (1727764429) mirror create pid 21730 mirror split pid 21731 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4827: 21730 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4827: 21731 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 02:34:52 (1727764492) Seed set to 4493 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1727764493.818383 write 0x010441 thru 0x0117fc (0x13bc bytes) 000005 1727764493.819894 read 0x0089fd thru 0x00c356 (0x395a bytes) 000006 1727764493.824960 write 0x01a767 thru 0x01fd9b (0x5635 bytes) truncating to largest ever: 0x3cf6a 000007 1727764493.832012 trunc from 0x01fd9c to 0x03cf69 (0x1d1ce bytes) 000008 1727764493.842904 read 0x0125f4 thru 0x0221f1 (0xfbfe bytes) 000009 1727764493.865625 trunc from 0x03cf6a to 0x02df09 (0xf060 bytes) 000012 1727764493.882181 mapread 0x01cf15 thru 0x02499e (0x7a8a bytes) 000013 1727764493.883118 mapwrite 0x031f24 thru 0x032553 (0x630 bytes) 000014 1727764494.038443 trunc from 0x032554 to 0x0380b5 (0x5b62 bytes) 000016 1727764494.051357 write 0x0150ef thru 0x01d20c (0x811e bytes) 000018 1727764494.052426 mapread 0x003593 thru 0x012354 (0xedc2 bytes) 000019 1727764494.055913 trunc from 0x0380b6 to 0x004579 (0x33b3c bytes) 000024 1727764494.086211 mapwrite 0x001d8e thru 0x00b671 (0x98e4 bytes) 000025 1727764494.188903 write 0x0231b4 thru 0x028eae (0x5cfb bytes) 000027 1727764494.192957 MIRROR_EXTEND to 2 mirrors 000029 1727764494.555725 read 0x010101 thru 0x019786 (0x9686 bytes) 000030 1727764494.574780 mapread 0x0072be thru 0x012184 (0xaec7 bytes) 000034 1727764494.614882 mapread 0x014835 thru 0x019b3a (0x5306 bytes) 000035 1727764494.619651 mapwrite 0x01aa2b thru 0x029d32 (0xf308 bytes) 000038 1727764494.801487 mapwrite 0x00e417 thru 0x01aac5 (0xc6af bytes) 000039 1727764494.892927 trunc from 0x029d33 to 0x010e65 (0x18ecd bytes) 000042 1727764494.933365 MIRROR_RESYNC 2 mirrors 000043 1727764495.332105 read 0x00f50b thru 0x010e65 (0x195b bytes) 000044 1727764495.342458 MIRROR_EXTEND to 3 mirrors 000045 1727764495.783142 trunc from 0x010e66 to 0x037d0e (0x26ea9 bytes) 000046 1727764495.860152 MIRROR_RESYNC 3 mirrors 000048 1727764496.372814 trunc from 0x037d0f to 0x0003eb (0x37923 bytes) 000049 1727764496.431114 mapread 0x000099 thru 0x0003eb (0x353 bytes) 000050 1727764496.435426 mapread 0x000312 thru 0x0003eb (0xda bytes) 000051 1727764496.436777 read 0x000221 thru 0x0003eb (0x1cb bytes) 000053 1727764496.437410 write 0x030bf9 thru 0x039880 (0x8c88 bytes) 000056 1727764496.439138 read 0x030cd6 thru 0x039880 (0x8bab bytes) 000057 1727764496.439620 read 0x018f40 thru 0x027a6d (0xeb2e bytes) 000058 1727764496.449417 trunc from 0x039881 to 0x038326 (0x155a bytes) 000064 1727764496.469260 mapread 0x0174a6 thru 0x01a0f2 (0x2c4d bytes) 000066 1727764496.478649 MIRROR_RESYNC 3 mirrors 000067 1727764496.944626 trunc from 0x038327 to 0x02cb6c (0xb7ba bytes) 000069 1727764497.046416 mapread 0x00384b thru 0x00e707 (0xaebd bytes) 000070 1727764497.064891 trunc from 0x02cb6d to 0x031234 (0x46c8 bytes) 000071 1727764497.078172 mapread 0x013981 thru 0x0174a6 (0x3b26 bytes) 000072 1727764497.086964 read 0x00d6bd thru 0x016608 (0x8f4c bytes) 000073 1727764497.088591 trunc from 0x031235 to 0x00080d (0x30a27 bytes) 000074 1727764497.103382 write 0x002550 thru 0x0055c0 (0x3071 bytes) 000075 1727764497.109283 MIRROR_EXTEND to 4 mirrors 000076 1727764497.550835 read 0x002266 thru 0x00491b (0x26b6 bytes) 000077 1727764497.559799 mapwrite 0x032fca thru 0x038acb (0x5b02 bytes) 000078 1727764497.728910 mapread 0x01721f thru 0x023a97 (0xc879 bytes) 000079 1727764497.743127 mapwrite 0x01ab22 thru 0x0256f8 (0xabd7 bytes) truncating to largest ever: 0x3d0cd 000082 1727764497.808949 trunc from 0x038acc to 0x03d0cc (0x4601 bytes) 000086 1727764497.828314 mapwrite 0x01863d thru 0x02456b (0xbf2f bytes) 000087 1727764497.893966 write 0x00ca54 thru 0x013163 (0x6710 bytes) 000088 1727764497.902753 trunc from 0x03d0cd to 0x002dce (0x3a2fe bytes) 000090 1727764497.918694 mapread 0x00225f thru 0x002dce (0xb70 bytes) 000091 1727764497.925767 mapread 0x000617 thru 0x002dce (0x27b8 bytes) 000092 1727764497.932040 mapread 0x002960 thru 0x002dce (0x46f bytes) 000095 1727764497.933323 trunc from 0x002dcf to 0x039a35 (0x36c67 bytes) 000097 1727764497.951422 read 0x01b9a0 thru 0x023d78 (0x83d9 bytes) 000098 1727764497.958792 trunc from 0x039a36 to 0x0058b3 (0x34182 bytes) 000100 1727764497.973574 read 0x004d94 thru 0x0058b3 (0xb20 bytes) 000101 1727764497.973984 read 0x002d03 thru 0x0041ca (0x14c8 bytes) 000102 1727764497.974378 trunc from 0x0058b4 to 0x0238ca (0x1e017 bytes) 000103 1727764497.989666 read 0x015703 thru 0x01a22f (0x4b2d bytes) 000104 1727764497.996007 read 0x007a9c thru 0x00956b (0x1ad0 bytes) 000108 1727764497.997316 mapwrite 0x01b95d thru 0x0235e3 (0x7c87 bytes) 000110 1727764498.052234 write 0x01b75c thru 0x0279b9 (0xc25e bytes) 000112 1727764498.053610 read 0x01934d thru 0x0279b9 (0xe66d bytes) 000113 1727764498.058035 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000114 1727764498.250359 read 0x0218b4 thru 0x0279b9 (0x6106 bytes) 000116 1727764498.256705 mapread 0x01b1b6 thru 0x021be2 (0x6a2d bytes) 000119 1727764498.273513 mapwrite 0x03460b thru 0x03ffff (0xb9f5 bytes) 000124 1727764498.358502 write 0x03f43e thru 0x03ffff (0xbc2 bytes) 000125 1727764498.359203 write 0x016b51 thru 0x017848 (0xcf8 bytes) 000129 1727764498.367797 mapread 0x01b370 thru 0x0226d1 (0x7362 bytes) 000131 1727764498.370228 write 0x0187b0 thru 0x0259e1 (0xd232 bytes) 000132 1727764498.371242 mapread 0x004bbc thru 0x00cc4c (0x8091 bytes) 000133 1727764498.382918 mapwrite 0x02f0fe thru 0x03c77a (0xd67d bytes) 000134 1727764498.455606 MIRROR_RESYNC 3 mirrors 000135 1727764498.936501 read 0x0322fc thru 0x03c5b3 (0xa2b8 bytes) 000140 1727764498.948435 write 0x01a7c4 thru 0x021e88 (0x76c5 bytes) 000142 1727764499.034002 trunc from 0x040000 to 0x0392b2 (0x6d4d bytes) 000143 1727764499.067246 mapread 0x02300c thru 0x0311d3 (0xe1c8 bytes) 000144 1727764499.087276 write 0x021064 thru 0x030910 (0xf8ad bytes) 000145 1727764499.088478 mapread 0x015acd thru 0x019e14 (0x4348 bytes) 000146 1727764499.096916 trunc from 0x0392b3 to 0x029728 (0xfb8a bytes) 000147 1727764499.116728 trunc from 0x029729 to 0x00cb5b (0x1cbcd bytes) 000148 1727764499.133124 write 0x025a5f thru 0x032b58 (0xd0fa bytes) 000150 1727764499.136751 read 0x020ce6 thru 0x02431a (0x3635 bytes) 000151 1727764499.147107 write 0x010ba5 thru 0x016ddb (0x6237 bytes) 000152 1727764499.147918 mapwrite 0x01b6fc thru 0x01cd33 (0x1638 bytes) 000153 1727764499.219699 write 0x01a161 thru 0x027e1f (0xdcbf bytes) 000154 1727764499.223839 write 0x02e79c thru 0x036ac3 (0x8328 bytes) 000155 1727764499.224727 mapwrite 0x00be5f thru 0x0116b9 (0x585b bytes) 000158 1727764499.305694 mapread 0x00d5ae thru 0x00d93b (0x38e bytes) 000159 1727764499.308200 trunc from 0x036ac4 to 0x0305a3 (0x6520 bytes) 000160 1727764499.324347 read 0x02131d thru 0x02b3d9 (0xa0bd bytes) 000161 1727764499.324862 mapwrite 0x0289f1 thru 0x035f1a (0xd52a bytes) 000162 1727764499.427978 mapread 0x0334a8 thru 0x035f1a (0x2a73 bytes) 000164 1727764499.429837 write 0x013faa thru 0x015f92 (0x1fe9 bytes) 000166 1727764499.431639 mapwrite 0x03c97a thru 0x03ffff (0x3686 bytes) 000168 1727764499.525514 read 0x00bca2 thru 0x01174d (0x5aac bytes) 000169 1727764499.525915 trunc from 0x040000 to 0x03ae8f (0x5170 bytes) 000170 1727764499.542584 trunc from 0x03ae90 to 0x004386 (0x36b09 bytes) 000171 1727764499.556890 trunc from 0x004387 to 0x0113d0 (0xd04a bytes) 000172 1727764499.573953 mapwrite 0x003eff thru 0x0094e0 (0x55e2 bytes) 000173 1727764499.640195 write 0x028dce thru 0x035bdf (0xce12 bytes) 000175 1727764499.642966 trunc from 0x035be0 to 0x03bf9e (0x63bf bytes) 000177 1727764499.660481 write 0x01cb96 thru 0x01f747 (0x2bb2 bytes) 000180 1727764499.668588 mapwrite 0x0316da thru 0x035d6f (0x4696 bytes) 000181 1727764499.725116 write 0x016317 thru 0x025335 (0xf01f bytes) 000183 1727764499.727114 trunc from 0x03bf9f to 0x031ab1 (0xa4ed bytes) 000184 1727764499.740340 MIRROR_EXTEND to 4 mirrors 000185 1727764500.256513 mapwrite 0x03f2e9 thru 0x03f575 (0x28d bytes) 000187 1727764500.406817 read 0x000c10 thru 0x00cac2 (0xbeb3 bytes) 000188 1727764500.449502 mapwrite 0x02a836 thru 0x035eb0 (0xb67b bytes) 000194 1727764500.561570 mapread 0x0390e0 thru 0x03a80b (0x172c bytes) 000195 1727764500.565880 MIRROR_EXTEND to 5 mirrors 000197 1727764500.895630 mapread 0x029cc6 thru 0x02c31b (0x2656 bytes) 000198 1727764500.910916 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000199 1727764501.118140 trunc from 0x03f576 to 0x017810 (0x27d65 bytes) 000200 1727764501.201801 trunc from 0x017811 to 0x018e4e (0x163e bytes) 000201 1727764501.226468 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000202 1727764501.465231 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x3e4bd 000207 1727764502.087425 trunc from 0x018e4f to 0x03e4bc (0x2566e bytes) 000208 1727764502.137611 write 0x02651d thru 0x032ab2 (0xc596 bytes) 000209 1727764502.150231 MIRROR_RESYNC 3 mirrors 000210 1727764502.568846 read 0x000aaa thru 0x000edd (0x434 bytes) 000213 1727764502.574179 trunc from 0x03e4bd to 0x00ba6b (0x32a51 bytes) 000214 1727764502.621979 MIRROR_EXTEND to 4 mirrors 000215 1727764502.986989 trunc from 0x00ba6c to 0x028ac9 (0x1d05e bytes) 000216 1727764503.045316 write 0x023f6f thru 0x0288c4 (0x4956 bytes) 000217 1727764503.053405 mapwrite 0x01325f thru 0x0172d1 (0x4073 bytes) 000218 1727764503.101881 mapwrite 0x034df5 thru 0x03aa72 (0x5c7e bytes) 000220 1727764503.212646 trunc from 0x03aa73 to 0x01ea54 (0x1c01e bytes) 000222 1727764503.228210 trunc from 0x01ea55 to 0x03a90f (0x1bebb bytes) 000223 1727764503.241293 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000224 1727764503.415507 trunc from 0x03a910 to 0x02f9ea (0xaf25 bytes) 000225 1727764503.436184 trunc from 0x02f9eb to 0x008dc0 (0x26c2a bytes) 000226 1727764503.457850 read 0x005fe2 thru 0x006c7b (0xc9a bytes) 000227 1727764503.463004 mapread 0x003d10 thru 0x008dc0 (0x50b1 bytes) 000228 1727764503.484669 read 0x003edb thru 0x008dc0 (0x4ee6 bytes) 000230 1727764503.486172 trunc from 0x008dc1 to 0x01d8c3 (0x14b03 bytes) 000231 1727764503.502393 trunc from 0x01d8c4 to 0x0096bd (0x14206 bytes) 000232 1727764503.526027 MIRROR_EXTEND to 4 mirrors 000233 1727764503.917431 mapwrite 0x01885d thru 0x028278 (0xfa1c bytes) 000235 1727764504.165011 write 0x033df7 thru 0x03ffff (0xc209 bytes) 000236 1727764504.166031 write 0x018104 thru 0x0233c5 (0xb2c2 bytes) 000238 1727764504.168316 mapread 0x01fb4c thru 0x02c859 (0xcd0e bytes) 000239 1727764504.181004 mapread 0x02194c thru 0x027b6b (0x6220 bytes) 000240 1727764504.188754 MIRROR_RESYNC 4 mirrors 000241 1727764504.632399 write 0x0203b9 thru 0x02dd67 (0xd9af bytes) 000243 1727764504.690962 mapwrite 0x02441c thru 0x02a772 (0x6357 bytes) 000244 1727764504.771105 read 0x02d4bc thru 0x03351a (0x605f bytes) 000249 1727764504.789852 mapread 0x023b6e thru 0x0318ef (0xdd82 bytes) 000250 1727764504.790986 mapwrite 0x036f52 thru 0x03ffff (0x90ae bytes) 000252 1727764504.863315 trunc from 0x040000 to 0x03a150 (0x5eaf bytes) 000253 1727764504.892971 mapread 0x019e65 thru 0x024223 (0xa3bf bytes) 000254 1727764504.897818 mapread 0x004c50 thru 0x004e0f (0x1c0 bytes) 000255 1727764504.906563 MIRROR_EXTEND to 5 mirrors 000256 1727764505.266574 MIRROR_RESYNC 5 mirrors 000257 1727764505.832855 trunc from 0x03a151 to 0x030d5a (0x93f6 bytes) 000262 1727764505.914377 trunc from 0x030d5b to 0x01cef3 (0x13e67 bytes) 000264 1727764505.932455 MIRROR_RESYNC 5 mirrors 000265 1727764506.472888 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000266 1727764506.659175 trunc from 0x01cef4 to 0x02e254 (0x11361 bytes) 000268 1727764506.742342 trunc from 0x02e255 to 0x01cc85 (0x115cf bytes) 000269 1727764506.762491 mapwrite 0x02bc59 thru 0x0324a6 (0x684e bytes) 000270 1727764506.854021 mapwrite 0x028aae thru 0x033f0f (0xb462 bytes) 000271 1727764506.970209 read 0x024bc3 thru 0x02ca6b (0x7ea9 bytes) 000272 1727764506.986620 MIRROR_EXTEND to 5 mirrors 000277 1727764507.383048 mapread 0x0157bf thru 0x01befc (0x673e bytes) 000279 1727764507.399847 write 0x00fcd0 thru 0x01f7d9 (0xfb0a bytes) 000281 1727764507.459748 mapread 0x020dd4 thru 0x02be12 (0xb03f bytes) 000282 1727764507.476141 mapread 0x032b90 thru 0x033f0f (0x1380 bytes) 000283 1727764507.478264 read 0x033788 thru 0x033f0f (0x788 bytes) 000284 1727764507.478686 write 0x02467e thru 0x031e25 (0xd7a8 bytes) 000285 1727764507.479598 mapwrite 0x038117 thru 0x03ffff (0x7ee9 bytes) 000288 1727764507.604303 read 0x03fc49 thru 0x03ffff (0x3b7 bytes) 000289 1727764507.605546 mapwrite 0x005f57 thru 0x01073e (0xa7e8 bytes) 000290 1727764507.687569 mapwrite 0x012b21 thru 0x0143f9 (0x18d9 bytes) 000291 1727764507.784947 read 0x022ab8 thru 0x02a6de (0x7c27 bytes) 000295 1727764507.788287 read 0x0193cc thru 0x01cfd0 (0x3c05 bytes) 000296 1727764507.788678 trunc from 0x040000 to 0x010d7d (0x2f282 bytes) 000297 1727764507.800722 mapwrite 0x01e224 thru 0x020fc9 (0x2da6 bytes) 000298 1727764507.931939 mapread 0x01e88e thru 0x020fc9 (0x273c bytes) 000299 1727764507.934432 read 0x01fe43 thru 0x020fc9 (0x1187 bytes) 000301 1727764507.941274 mapread 0x00aa55 thru 0x015129 (0xa6d5 bytes) 000304 1727764507.956215 MIRROR_RESYNC 5 mirrors 000305 1727764508.532189 MIRROR_RESYNC 5 mirrors 000306 1727764508.687908 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000307 1727764508.949202 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000308 1727764509.142624 write 0x03492a thru 0x03d212 (0x88e9 bytes) 000309 1727764509.193735 mapread 0x0190c4 thru 0x01ab85 (0x1ac2 bytes) 000310 1727764509.200862 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000316 1727764509.339616 MIRROR_EXTEND to 3 mirrors 000317 1727764509.734340 write 0x024edc thru 0x02e760 (0x9885 bytes) 000318 1727764509.795939 write 0x030ece thru 0x03fc80 (0xedb3 bytes) 000319 1727764509.807593 mapwrite 0x01933e thru 0x01fc12 (0x68d5 bytes) 000320 1727764509.868163 mapread 0x0212fd thru 0x028823 (0x7527 bytes) 000321 1727764509.876223 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000324 1727764510.004355 read 0x0229c9 thru 0x02e8ae (0xbee6 bytes) 000325 1727764510.014423 write 0x036259 thru 0x03d19d (0x6f45 bytes) 000328 1727764510.023025 trunc from 0x03fc81 to 0x0257cb (0x1a4b5 bytes) 000329 1727764510.034634 mapwrite 0x015ff0 thru 0x02463d (0xe64e bytes) 000330 1727764510.118121 MIRROR_RESYNC 2 mirrors 000336 1727764510.511535 mapread 0x002d37 thru 0x011dde (0xf0a8 bytes) 000337 1727764510.536268 write 0x006582 thru 0x0138c8 (0xd347 bytes) 000339 1727764510.613029 MIRROR_RESYNC 2 mirrors 000340 1727764510.906611 mapread 0x008037 thru 0x011830 (0x97fa bytes) 000341 1727764510.926439 read 0x001083 thru 0x00e8e9 (0xd867 bytes) 000342 1727764510.940602 MIRROR_EXTEND to 3 mirrors 000343 1727764511.345312 read 0x006353 thru 0x010e4f (0xaafd bytes) 000345 1727764511.351020 trunc from 0x0257cc to 0x009fc0 (0x1b80b bytes) 000346 1727764511.433792 mapread 0x0045c2 thru 0x009fc0 (0x59ff bytes) 000347 1727764511.452861 read 0x008488 thru 0x009fc0 (0x1b39 bytes) 000348 1727764511.453312 trunc from 0x009fc1 to 0x02a9f4 (0x20a34 bytes) 000352 1727764511.482938 write 0x03e02b thru 0x03ffff (0x1fd5 bytes) 000360 1727764511.486459 write 0x02d222 thru 0x02e25b (0x103a bytes) 000361 1727764511.509723 read 0x01aff9 thru 0x022c32 (0x7c3a bytes) 000363 1727764511.515026 MIRROR_RESYNC 3 mirrors 000365 1727764511.952769 mapread 0x01bb3f thru 0x01d491 (0x1953 bytes) 000369 1727764511.977132 mapread 0x0298f9 thru 0x033b63 (0xa26b bytes) 000370 1727764512.002772 MIRROR_EXTEND to 4 mirrors 000371 1727764512.347779 mapwrite 0x02750a thru 0x02db41 (0x6638 bytes) 000372 1727764512.480786 read 0x00826b thru 0x00cba8 (0x493e bytes) 000373 1727764512.492420 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000375 1727764512.664058 mapread 0x02c54c thru 0x0302ea (0x3d9f bytes) 000376 1727764512.678956 trunc from 0x040000 to 0x00e885 (0x3177a bytes) 000384 1727764512.693585 mapwrite 0x01d884 thru 0x01f0dd (0x185a bytes) 000387 1727764512.768904 trunc from 0x01f0de to 0x031e8c (0x12daf bytes) 000388 1727764512.791935 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000392 1727764512.919997 mapwrite 0x01e9ee thru 0x028ad6 (0xa0e9 bytes) 000393 1727764512.993967 MIRROR_EXTEND to 3 mirrors 000394 1727764513.309433 MIRROR_EXTEND to 4 mirrors 000396 1727764513.718722 write 0x015c6a thru 0x022093 (0xc42a bytes) 000398 1727764513.786112 read 0x0196fb thru 0x027f24 (0xe82a bytes) 000402 1727764513.794657 read 0x026460 thru 0x031e8c (0xba2d bytes) 000403 1727764513.806080 write 0x025c3c thru 0x034ddc (0xf1a1 bytes) 000407 1727764513.815966 MIRROR_RESYNC 4 mirrors truncating to largest ever: 0x3eb09 000408 1727764514.380271 trunc from 0x034ddd to 0x03eb08 (0x9d2c bytes) 000410 1727764514.439741 read 0x034e88 thru 0x03eb08 (0x9c81 bytes) 000411 1727764514.443778 read 0x03def7 thru 0x03eb08 (0xc12 bytes) 000413 1727764514.445863 read 0x027072 thru 0x033eb2 (0xce41 bytes) 000415 1727764514.456667 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000416 1727764514.585312 MIRROR_RESYNC 3 mirrors 000417 1727764514.954390 mapwrite 0x038a12 thru 0x03ffff (0x75ee bytes) 000418 1727764515.099570 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000421 1727764515.248426 trunc from 0x040000 to 0x002a5e (0x3d5a1 bytes) 000423 1727764515.263252 MIRROR_RESYNC 2 mirrors 000424 1727764515.646686 mapwrite 0x019204 thru 0x02855b (0xf358 bytes) 000425 1727764515.807874 read 0x01f89c thru 0x0234c0 (0x3c25 bytes) 000426 1727764515.808323 mapwrite 0x01a419 thru 0x02645e (0xc046 bytes) 000427 1727764515.866640 trunc from 0x02855c to 0x00b507 (0x1d054 bytes) 000428 1727764515.890956 write 0x02fca9 thru 0x03b072 (0xb3ca bytes) 000429 1727764515.892348 mapwrite 0x015e46 thru 0x018a54 (0x2c0f bytes) 000430 1727764515.971686 mapwrite 0x00a690 thru 0x00dc69 (0x35da bytes) 000433 1727764516.031281 mapread 0x0191ea thru 0x01be99 (0x2cb0 bytes) 000434 1727764516.031854 trunc from 0x03b073 to 0x026738 (0x1493a bytes) 000435 1727764516.053497 MIRROR_EXTEND to 3 mirrors 000436 1727764516.372105 mapwrite 0x000312 thru 0x00d92e (0xd61d bytes) 000437 1727764516.535247 mapread 0x008e40 thru 0x0176db (0xe89c bytes) 000438 1727764516.554504 trunc from 0x026739 to 0x0037bd (0x22f7b bytes) 000439 1727764516.577872 mapwrite 0x0003ba thru 0x005834 (0x547b bytes) 000440 1727764516.652249 write 0x001ad6 thru 0x00a939 (0x8e64 bytes) 000442 1727764516.653340 write 0x0150a1 thru 0x01a5dd (0x553d bytes) 000443 1727764516.653967 read 0x001221 thru 0x00856b (0x734b bytes) 000445 1727764516.654586 mapread 0x004572 thru 0x010a47 (0xc4d6 bytes) 000447 1727764516.660843 write 0x022f73 thru 0x02bc9c (0x8d2a bytes) 000451 1727764516.665221 mapread 0x012f6d thru 0x0209f0 (0xda84 bytes) 000452 1727764516.681069 MIRROR_RESYNC 3 mirrors 000453 1727764517.052126 trunc from 0x02bc9d to 0x03112e (0x5492 bytes) 000455 1727764517.127595 mapread 0x010e58 thru 0x01ffff (0xf1a8 bytes) 000457 1727764517.164728 trunc from 0x03112f to 0x00f172 (0x21fbc bytes) 000458 1727764517.201813 mapwrite 0x0149ac thru 0x01ae68 (0x64bd bytes) 000459 1727764517.321650 MIRROR_EXTEND to 4 mirrors 000462 1727764517.701882 mapwrite 0x01d822 thru 0x01eb68 (0x1347 bytes) 000463 1727764517.847200 MIRROR_RESYNC 4 mirrors 000464 1727764518.321543 mapwrite 0x00c085 thru 0x013a03 (0x797f bytes) 000467 1727764518.483805 mapread 0x00b0ea thru 0x00c301 (0x1218 bytes) 000470 1727764518.484986 write 0x007775 thru 0x011583 (0x9e0f bytes) 000471 1727764518.491308 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000472 1727764518.682155 mapread 0x01ac08 thru 0x01b465 (0x85e bytes) 000473 1727764518.698834 read 0x013d8b thru 0x01eb68 (0xadde bytes) 000475 1727764518.711927 MIRROR_RESYNC 3 mirrors 000476 1727764519.127722 write 0x01bcaf thru 0x026983 (0xacd5 bytes) 000479 1727764519.186177 MIRROR_EXTEND to 4 mirrors 000482 1727764519.550115 read 0x00270e thru 0x011954 (0xf247 bytes) 000483 1727764519.561047 read 0x0239fc thru 0x026983 (0x2f88 bytes) 000484 1727764519.570411 write 0x00ae39 thru 0x01ab29 (0xfcf1 bytes) 000485 1727764519.628444 mapread 0x025c7b thru 0x026983 (0xd09 bytes) 000486 1727764519.636979 mapwrite 0x03c3c1 thru 0x03c632 (0x272 bytes) 000487 1727764519.733612 mapread 0x035feb thru 0x03c632 (0x6648 bytes) 000488 1727764519.734247 read 0x025645 thru 0x034980 (0xf33c bytes) 000490 1727764519.735052 read 0x01a5a7 thru 0x01e897 (0x42f1 bytes) 000491 1727764519.745276 write 0x007a44 thru 0x0096cb (0x1c88 bytes) 000492 1727764519.746552 read 0x02ded4 thru 0x03c632 (0xe75f bytes) 000493 1727764519.746991 write 0x027cbd thru 0x032672 (0xa9b6 bytes) 000495 1727764519.748268 mapread 0x031148 thru 0x03c632 (0xb4eb bytes) 000496 1727764519.749075 write 0x02b668 thru 0x035758 (0xa0f1 bytes) 000499 1727764519.750704 mapwrite 0x039c87 thru 0x03aecc (0x1246 bytes) 000500 1727764519.808305 write 0x011b8d thru 0x01de3f (0xc2b3 bytes) 000501 1727764519.810908 trunc from 0x03c633 to 0x000374 (0x3c2be bytes) 000502 1727764519.836708 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000503 1727764519.986595 trunc from 0x000375 to 0x033c85 (0x33911 bytes) 000504 1727764519.995635 write 0x0343ae thru 0x03ffff (0xbc52 bytes) 000505 1727764519.996706 trunc from 0x040000 to 0x035b9d (0xa462 bytes) 000510 1727764520.011476 read 0x015be7 thru 0x0241bb (0xe5d5 bytes) 000511 1727764520.017211 mapwrite 0x0123fd thru 0x0134dd (0x10e1 bytes) 000512 1727764520.073977 write 0x029a6e thru 0x035d89 (0xc31c bytes) 000513 1727764520.082103 write 0x0140e0 thru 0x020522 (0xc443 bytes) 000514 1727764520.089388 read 0x028600 thru 0x029dce (0x17cf bytes) 000516 1727764520.092386 read 0x0226e5 thru 0x026291 (0x3bad bytes) 000518 1727764520.093905 mapwrite 0x0054ce thru 0x011fac (0xcadf bytes) 000519 1727764520.155564 write 0x037c76 thru 0x03d490 (0x581b bytes) 000520 1727764520.156549 trunc from 0x03d491 to 0x031423 (0xc06d bytes) 000521 1727764520.173137 trunc from 0x031424 to 0x017af0 (0x19933 bytes) 000523 1727764520.182869 mapwrite 0x01ea21 thru 0x02cef3 (0xe4d3 bytes) 000524 1727764520.304792 trunc from 0x02cef4 to 0x00577c (0x27777 bytes) 000525 1727764520.323987 mapread 0x0052e0 thru 0x00577c (0x49d bytes) 000528 1727764520.325374 write 0x006690 thru 0x00b7c4 (0x5135 bytes) 000532 1727764520.329433 mapread 0x005a10 thru 0x005fe0 (0x5d1 bytes) 000533 1727764520.331679 read 0x004c8e thru 0x0090d0 (0x4443 bytes) 000534 1727764520.332079 mapwrite 0x03b525 thru 0x03ffff (0x4adb bytes) 000535 1727764520.417205 mapread 0x03cdba thru 0x03ffff (0x3246 bytes) 000536 1727764520.418047 read 0x016825 thru 0x024229 (0xda05 bytes) 000537 1727764520.418887 write 0x03d780 thru 0x03ffff (0x2880 bytes) 000538 1727764520.419837 read 0x024a19 thru 0x03336e (0xe956 bytes) 000539 1727764520.420853 trunc from 0x040000 to 0x01267f (0x2d980 bytes) 000540 1727764520.433866 write 0x005745 thru 0x009e26 (0x46e2 bytes) 000541 1727764520.437478 MIRROR_RESYNC 3 mirrors 000542 1727764520.762057 write 0x00ddf5 thru 0x01d8af (0xfabb bytes) 000543 1727764520.825121 mapwrite 0x00c6a0 thru 0x0152a6 (0x8c07 bytes) 000544 1727764520.896786 mapread 0x001303 thru 0x0060e7 (0x4de5 bytes) 000545 1727764520.910587 read 0x0187bc thru 0x01d8af (0x50f4 bytes) 000547 1727764520.913147 trunc from 0x01d8b0 to 0x02f567 (0x11cb8 bytes) 000550 1727764520.938291 mapwrite 0x0117cb thru 0x01c6e3 (0xaf19 bytes) 000552 1727764521.004110 mapwrite 0x02d457 thru 0x039c21 (0xc7cb bytes) 000555 1727764521.092222 mapwrite 0x026e67 thru 0x033425 (0xc5bf bytes) 000556 1727764521.138301 read 0x03252f thru 0x0344f7 (0x1fc9 bytes) 000557 1727764521.138726 trunc from 0x039c22 to 0x02b0f7 (0xeb2a bytes) 000558 1727764521.155816 write 0x033680 thru 0x03da25 (0xa3a6 bytes) 000559 1727764521.167614 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000561 1727764521.287717 mapread 0x0274e4 thru 0x028823 (0x1340 bytes) 000562 1727764521.294632 mapread 0x0063f0 thru 0x014796 (0xe3a7 bytes) 000563 1727764521.302628 read 0x034146 thru 0x03a85d (0x6718 bytes) 000565 1727764521.306981 read 0x02321e thru 0x0259d5 (0x27b8 bytes) 000566 1727764521.317935 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000568 1727764521.401582 write 0x031cc7 thru 0x03cc11 (0xaf4b bytes) 000569 1727764521.409614 mapwrite 0x02faa0 thru 0x03a18f (0xa6f0 bytes) 000570 1727764521.461957 write 0x013e2c thru 0x023b18 (0xfced bytes) 000574 1727764521.473046 read 0x02e9c7 thru 0x039ea3 (0xb4dd bytes) 000576 1727764521.474177 mapread 0x021c8e thru 0x026271 (0x45e4 bytes) 000581 1727764521.479979 mapread 0x0072b5 thru 0x013a36 (0xc782 bytes) 000582 1727764521.487252 read 0x00d77d thru 0x010e00 (0x3684 bytes) 000584 1727764521.488784 read 0x019cb2 thru 0x01e9dd (0x4d2c bytes) 000585 1727764521.489812 trunc from 0x03da26 to 0x00bc73 (0x31db2 bytes) 000586 1727764521.505588 MIRROR_EXTEND to 2 mirrors 000588 1727764521.816205 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000589 1727764521.888936 write 0x035981 thru 0x03a446 (0x4ac6 bytes) 000590 1727764521.896753 read 0x018d70 thru 0x023bcb (0xae5c bytes) 000591 1727764521.901558 read 0x008579 thru 0x008bbe (0x646 bytes) 000593 1727764521.909882 mapwrite 0x0338f2 thru 0x035134 (0x1843 bytes) 000594 1727764521.953746 read 0x013ebd thru 0x021729 (0xd86d bytes) 000596 1727764521.954500 mapread 0x0372c0 thru 0x03a446 (0x3187 bytes) 000597 1727764521.955040 mapwrite 0x0065d2 thru 0x010f26 (0xa955 bytes) 000598 1727764522.003126 mapread 0x01e06c thru 0x02a99a (0xc92f bytes) 000599 1727764522.003736 trunc from 0x03a447 to 0x013cd5 (0x26771 bytes) 000601 1727764522.030854 write 0x01b82c thru 0x029f0f (0xe6e4 bytes) 000602 1727764522.031864 write 0x031bd9 thru 0x0390e6 (0x750e bytes) 000604 1727764522.033135 read 0x008df6 thru 0x008e00 (0xb bytes) 000609 1727764522.035592 trunc from 0x0390e7 to 0x01e892 (0x1a854 bytes) 000610 1727764522.050713 read 0x00a7d1 thru 0x00ecfa (0x452a bytes) 000611 1727764522.051289 trunc from 0x01e893 to 0x00b02a (0x13868 bytes) 000612 1727764522.062108 trunc from 0x00b02b to 0x014840 (0x9816 bytes) 000614 1727764522.074916 mapwrite 0x0312fc thru 0x03f08b (0xdd90 bytes) 000616 1727764522.165668 write 0x03b823 thru 0x03ffff (0x47dd bytes) 000617 1727764522.166995 mapwrite 0x01faab thru 0x027993 (0x7ee9 bytes) 000620 1727764522.226885 mapread 0x018a63 thru 0x024d38 (0xc2d6 bytes) 000624 1727764522.230670 read 0x0036ed thru 0x00ba51 (0x8365 bytes) 000625 1727764522.231077 mapread 0x0139af thru 0x015f8d (0x25df bytes) 000628 1727764522.232364 mapread 0x00b6ee thru 0x015c94 (0xa5a7 bytes) 000632 1727764522.236286 mapwrite 0x026d38 thru 0x030d65 (0xa02e bytes) 000634 1727764522.291235 trunc from 0x040000 to 0x02b3ac (0x14c53 bytes) 000636 1727764522.303762 read 0x002095 thru 0x010688 (0xe5f4 bytes) 000640 1727764522.305734 trunc from 0x02b3ad to 0x02f26b (0x3ebf bytes) 000641 1727764522.317330 mapwrite 0x00b7fc thru 0x016d9a (0xb59f bytes) 000642 1727764522.370358 read 0x0142c8 thru 0x016cf9 (0x2a32 bytes) 000643 1727764522.370745 write 0x02abdd thru 0x037f18 (0xd33c bytes) 000648 1727764522.372741 trunc from 0x037f19 to 0x01b7db (0x1c73d bytes) 000649 1727764522.390705 read 0x009cd2 thru 0x00ce7e (0x31ad bytes) 000651 1727764522.391343 read 0x00fd6f thru 0x01b623 (0xb8b5 bytes) 000657 1727764522.396687 read 0x013ec9 thru 0x015b68 (0x1ca0 bytes) 000658 1727764522.398072 mapread 0x0175e1 thru 0x01b7db (0x41fb bytes) 000659 1727764522.405759 MIRROR_EXTEND to 2 mirrors 000661 1727764522.706043 mapwrite 0x02ff09 thru 0x03ae77 (0xaf6f bytes) 000664 1727764522.866014 MIRROR_RESYNC 2 mirrors 000666 1727764523.147358 read 0x033c75 thru 0x03824b (0x45d7 bytes) 000667 1727764523.154884 write 0x03a42f thru 0x03ffff (0x5bd1 bytes) 000669 1727764523.205114 write 0x01d73e thru 0x01f351 (0x1c14 bytes) 000671 1727764523.221427 trunc from 0x040000 to 0x00ffcc (0x30033 bytes) 000673 1727764523.233281 trunc from 0x00ffcd to 0x01ac8f (0xacc3 bytes) 000674 1727764523.252037 mapwrite 0x0285e6 thru 0x02ad13 (0x272e bytes) truncating to largest ever: 0x3f3f9 000677 1727764523.333230 trunc from 0x02ad14 to 0x03f3f8 (0x146e5 bytes) 000678 1727764523.347297 read 0x002535 thru 0x008c34 (0x6700 bytes) 000679 1727764523.358129 trunc from 0x03f3f9 to 0x036e2c (0x85cc bytes) 000681 1727764523.376156 read 0x01c31a thru 0x025a7b (0x9762 bytes) 000683 1727764523.378457 mapread 0x02fe62 thru 0x030ea1 (0x1040 bytes) 000684 1727764523.382161 mapwrite 0x025439 thru 0x02d62e (0x81f6 bytes) 000685 1727764523.436352 write 0x00d814 thru 0x018515 (0xad02 bytes) 000686 1727764523.444336 MIRROR_EXTEND to 3 mirrors 000687 1727764523.812217 write 0x004548 thru 0x0083bd (0x3e76 bytes) 000688 1727764523.882733 read 0x0363bb thru 0x036e2c (0xa72 bytes) 000691 1727764523.888127 trunc from 0x036e2d to 0x035e27 (0x1005 bytes) 000693 1727764523.916580 trunc from 0x035e28 to 0x033657 (0x27d0 bytes) 000696 1727764523.929662 trunc from 0x033658 to 0x03d639 (0x9fe2 bytes) 000697 1727764523.942008 write 0x02bc69 thru 0x034f6b (0x9303 bytes) 000698 1727764523.951629 write 0x0232cb thru 0x032161 (0xee97 bytes) 000699 1727764523.963150 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000700 1727764524.081234 mapwrite 0x02efce thru 0x0388be (0x98f1 bytes) 000703 1727764524.138299 read 0x00c944 thru 0x01025e (0x391b bytes) 000705 1727764524.148372 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000707 1727764524.234308 read 0x003c10 thru 0x00fec3 (0xc2b4 bytes) 000709 1727764524.242983 mapread 0x00ef70 thru 0x014e38 (0x5ec9 bytes) 000710 1727764524.256071 write 0x017f92 thru 0x01b2e6 (0x3355 bytes) 000711 1727764524.261927 write 0x028ad6 thru 0x0325af (0x9ada bytes) 000713 1727764524.275316 mapread 0x0008bd thru 0x006b30 (0x6274 bytes) 000715 1727764524.288065 write 0x03828f thru 0x03ffff (0x7d71 bytes) 000717 1727764524.292634 mapwrite 0x00d323 thru 0x019542 (0xc220 bytes) 000720 1727764524.362821 write 0x01c157 thru 0x01fac9 (0x3973 bytes) 000727 1727764524.379033 write 0x035cf1 thru 0x03ffff (0xa30f bytes) 000730 1727764524.381708 write 0x02a242 thru 0x0350d1 (0xae90 bytes) 000731 1727764524.382503 trunc from 0x040000 to 0x039973 (0x668c bytes) 000733 1727764524.405322 mapread 0x02349f thru 0x02cf81 (0x9ae3 bytes) 000735 1727764524.409632 mapwrite 0x00e33c thru 0x011a67 (0x372c bytes) 000736 1727764524.468830 mapwrite 0x005ec1 thru 0x015918 (0xfa58 bytes) 000738 1727764524.520642 trunc from 0x039974 to 0x01bee0 (0x1da93 bytes) 000740 1727764524.538166 mapwrite 0x00796d thru 0x009a1c (0x20b0 bytes) 000741 1727764524.596886 trunc from 0x01bee1 to 0x002f49 (0x18f97 bytes) 000742 1727764524.606566 trunc from 0x002f4a to 0x015df1 (0x12ea8 bytes) 000746 1727764524.618614 trunc from 0x015df2 to 0x0092f4 (0xcafd bytes) 000747 1727764524.638347 read 0x00110c thru 0x00677d (0x5672 bytes) 000748 1727764524.651242 mapread 0x001899 thru 0x0064c5 (0x4c2d bytes) 000749 1727764524.652613 mapread 0x0014e1 thru 0x0092f4 (0x7e14 bytes) 000754 1727764524.657648 mapread 0x0022c0 thru 0x0092f4 (0x7035 bytes) 000755 1727764524.660197 read 0x007390 thru 0x0092f4 (0x1f65 bytes) 000758 1727764524.662843 mapread 0x001682 thru 0x009182 (0x7b01 bytes) 000760 1727764524.673666 MIRROR_EXTEND to 2 mirrors 000762 1727764525.025476 mapwrite 0x004730 thru 0x00d72c (0x8ffd bytes) 000763 1727764525.156687 MIRROR_RESYNC 2 mirrors 000764 1727764525.493948 write 0x03d058 thru 0x03d8ab (0x854 bytes) 000765 1727764525.548774 write 0x039221 thru 0x03ffff (0x6ddf bytes) 000767 1727764525.555155 write 0x0019bb thru 0x00c273 (0xa8b9 bytes) 000768 1727764525.585867 read 0x011dd6 thru 0x01af8f (0x91ba bytes) 000769 1727764525.600072 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000772 1727764525.714558 read 0x02649c thru 0x02aeb3 (0x4a18 bytes) 000773 1727764525.724510 trunc from 0x040000 to 0x012f8e (0x2d071 bytes) 000774 1727764525.750139 trunc from 0x012f8f to 0x005da2 (0xd1ec bytes) 000779 1727764525.768863 read 0x003889 thru 0x005da2 (0x251a bytes) 000780 1727764525.780356 mapread 0x0053dc thru 0x005da2 (0x9c7 bytes) 000781 1727764525.784115 mapwrite 0x03f877 thru 0x03fded (0x577 bytes) 000782 1727764525.872009 mapwrite 0x000537 thru 0x00a178 (0x9c42 bytes) 000785 1727764525.940613 read 0x0327f0 thru 0x03fded (0xd5fe bytes) 000787 1727764525.943997 mapread 0x0147ac thru 0x0153f4 (0xc49 bytes) 000788 1727764525.945731 trunc from 0x03fdee to 0x006d14 (0x390d9 bytes) 000789 1727764525.965132 read 0x00279b thru 0x006d14 (0x457a bytes) 000794 1727764525.970966 trunc from 0x006d15 to 0x032c53 (0x2bf3f bytes) 000797 1727764525.989360 read 0x032b59 thru 0x032c53 (0xfb bytes) 000801 1727764526.011087 write 0x01d76c thru 0x027f1d (0xa7b2 bytes) 000802 1727764526.015138 trunc from 0x032c54 to 0x002bac (0x300a7 bytes) 000804 1727764526.046563 mapwrite 0x014f37 thru 0x018753 (0x381d bytes) 000805 1727764526.170578 MIRROR_EXTEND to 2 mirrors 000808 1727764526.561254 write 0x016c74 thru 0x01923a (0x25c7 bytes) 000810 1727764526.625779 read 0x006e6c thru 0x00f7d0 (0x8965 bytes) 000811 1727764526.636373 read 0x0140fe thru 0x01923a (0x513d bytes) 000812 1727764526.642406 write 0x0105c1 thru 0x01a6dd (0xa11d bytes) 000814 1727764526.649227 MIRROR_RESYNC 2 mirrors 000815 1727764527.005456 trunc from 0x01a6de to 0x023283 (0x8ba6 bytes) 000816 1727764527.116356 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000817 1727764527.211481 mapwrite 0x0157a7 thru 0x01e278 (0x8ad2 bytes) 000818 1727764527.277195 read 0x01d9f0 thru 0x023283 (0x5894 bytes) 000819 1727764527.285705 mapwrite 0x01fd36 thru 0x025436 (0x5701 bytes) 000821 1727764527.366334 read 0x0100c2 thru 0x018013 (0x7f52 bytes) 000823 1727764527.367222 mapwrite 0x00c4e0 thru 0x00d61d (0x113e bytes) 000825 1727764527.431379 mapread 0x0101e0 thru 0x0154a8 (0x52c9 bytes) 000826 1727764527.433907 read 0x020519 thru 0x025436 (0x4f1e bytes) 000829 1727764527.434982 mapwrite 0x003ce5 thru 0x00edd7 (0xb0f3 bytes) 000830 1727764527.507682 mapwrite 0x00bbbe thru 0x01a5af (0xe9f2 bytes) 000832 1727764527.566838 trunc from 0x025437 to 0x01fc64 (0x57d2 bytes) 000834 1727764527.581115 read 0x0109e6 thru 0x01180e (0xe29 bytes) 000836 1727764527.582893 trunc from 0x01fc65 to 0x039af1 (0x19e8d bytes) 000837 1727764527.594888 read 0x036012 thru 0x039af1 (0x3ae0 bytes) 000839 1727764527.607003 mapwrite 0x01a571 thru 0x01c924 (0x23b4 bytes) 000840 1727764527.664139 read 0x01fcf1 thru 0x02076b (0xa7b bytes) 000842 1727764527.665201 mapread 0x031894 thru 0x036e89 (0x55f6 bytes) 000847 1727764527.666958 trunc from 0x039af2 to 0x037265 (0x288c bytes) 000849 1727764527.675556 read 0x023f8b thru 0x02a119 (0x618f bytes) 000850 1727764527.677059 mapwrite 0x037d84 thru 0x03ffff (0x827c bytes) 000853 1727764527.745764 trunc from 0x040000 to 0x028ddf (0x17220 bytes) 000854 1727764527.765094 mapwrite 0x027cc6 thru 0x030f96 (0x92d1 bytes) 000855 1727764527.858158 mapwrite 0x0331e6 thru 0x033756 (0x571 bytes) 000858 1727764527.918893 mapwrite 0x03e47b thru 0x03ffff (0x1b85 bytes) 000859 1727764527.994234 mapwrite 0x008f53 thru 0x0113b2 (0x8460 bytes) 000861 1727764528.039572 mapread 0x01c324 thru 0x01d989 (0x1666 bytes) 000864 1727764528.040764 mapread 0x011fdb thru 0x012290 (0x2b6 bytes) 000865 1727764528.041256 mapwrite 0x004a50 thru 0x009104 (0x46b5 bytes) 000866 1727764528.088514 read 0x03964f thru 0x03ffff (0x69b1 bytes) 000867 1727764528.093207 write 0x01cea7 thru 0x01dd9d (0xef7 bytes) 000868 1727764528.095410 mapread 0x03c416 thru 0x03ffff (0x3bea bytes) 000870 1727764528.096951 write 0x005079 thru 0x00bb09 (0x6a91 bytes) truncating to largest ever: 0x3f81d 000871 1727764528.097736 trunc from 0x040000 to 0x03f81c (0x7e3 bytes) 000874 1727764528.111109 mapwrite 0x01c5ac thru 0x025a08 (0x945d bytes) 000877 1727764528.186795 MIRROR_EXTEND to 2 mirrors 000879 1727764528.534606 trunc from 0x03f81d to 0x0280b4 (0x17768 bytes) 000881 1727764528.621797 mapwrite 0x0169a7 thru 0x01a33a (0x3994 bytes) 000882 1727764528.719537 trunc from 0x0280b5 to 0x035734 (0xd680 bytes) 000884 1727764528.736135 mapwrite 0x029f93 thru 0x034a3f (0xaaad bytes) 000887 1727764528.804920 write 0x0120bf thru 0x018f84 (0x6ec6 bytes) 000888 1727764528.806505 write 0x00b4e0 thru 0x019521 (0xe042 bytes) 000889 1727764528.817259 MIRROR_EXTEND to 3 mirrors 000890 1727764529.335002 write 0x0312bf thru 0x03e006 (0xcd48 bytes) 000891 1727764529.395594 read 0x033fd8 thru 0x03e006 (0xa02f bytes) 000893 1727764529.401024 MIRROR_EXTEND to 4 mirrors 000896 1727764529.746127 mapwrite 0x013de4 thru 0x01e708 (0xa925 bytes) 000899 1727764529.936771 write 0x02c792 thru 0x02ee5f (0x26ce bytes) 000901 1727764529.945123 write 0x020068 thru 0x02898e (0x8927 bytes) 000902 1727764529.950953 MIRROR_EXTEND to 5 mirrors truncating to largest ever: 0x3f841 000904 1727764530.433193 trunc from 0x03e007 to 0x03f840 (0x183a bytes) 000911 1727764530.508880 write 0x01fb7c thru 0x02e990 (0xee15 bytes) 000913 1727764530.520141 trunc from 0x03f841 to 0x03d003 (0x283d bytes) 000919 1727764530.538621 write 0x039bcd thru 0x03ffff (0x6433 bytes) 000921 1727764530.549132 trunc from 0x040000 to 0x007d31 (0x382ce bytes) 000922 1727764530.567365 MIRROR_EXTEND to 6 mirrors 000923 1727764530.925847 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000924 1727764531.089326 mapwrite 0x03c0cf thru 0x03ffff (0x3f31 bytes) 000925 1727764531.229800 write 0x013dc3 thru 0x01f623 (0xb861 bytes) 000929 1727764531.241031 trunc from 0x040000 to 0x02baba (0x14545 bytes) 000930 1727764531.253889 read 0x00580e thru 0x009379 (0x3b6c bytes) 000932 1727764531.265078 MIRROR_RESYNC 5 mirrors 000933 1727764531.821826 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000935 1727764531.993536 MIRROR_EXTEND to 5 mirrors 000936 1727764532.394848 mapread 0x016936 thru 0x01eb74 (0x823f bytes) 000939 1727764532.410907 mapwrite 0x03afda thru 0x03eb8a (0x3bb1 bytes) 000941 1727764532.655779 mapread 0x01d5db thru 0x029dec (0xc812 bytes) 000943 1727764532.682069 read 0x039b85 thru 0x03e2d8 (0x4754 bytes) 000944 1727764532.682578 mapread 0x02bb77 thru 0x0306e5 (0x4b6f bytes) 000946 1727764532.683428 mapread 0x0169b9 thru 0x025a30 (0xf078 bytes) 000949 1727764532.692024 trunc from 0x03eb8b to 0x03f706 (0xb7c bytes) 000952 1727764532.713365 mapwrite 0x03bdd2 thru 0x03ffff (0x422e bytes) 000953 1727764532.801699 mapread 0x038e3c thru 0x03ffff (0x71c4 bytes) 000954 1727764532.802316 trunc from 0x040000 to 0x001fe0 (0x3e01f bytes) 000955 1727764532.813895 read 0x0019cc thru 0x001fe0 (0x615 bytes) 000958 1727764532.820157 write 0x0186d6 thru 0x02429a (0xbbc5 bytes) 000960 1727764532.821552 mapwrite 0x02b65b thru 0x030894 (0x523a bytes) 000961 1727764532.889082 write 0x032791 thru 0x035026 (0x2896 bytes) 000964 1727764532.894576 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000966 1727764533.029370 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x380:0x0]: Invalid argument (22) 000968 1727764533.155729 mapread 0x01cca6 thru 0x0299e2 (0xcd3d bytes) 000969 1727764533.178479 write 0x003dbf thru 0x012e6e (0xf0b0 bytes) 000970 1727764533.199657 write 0x03c96b thru 0x03ffff (0x3695 bytes) 000971 1727764533.200302 read 0x032858 thru 0x03fcc3 (0xd46c bytes) 000972 1727764533.209350 trunc from 0x040000 to 0x02f4e5 (0x10b1a bytes) 000973 1727764533.221559 trunc from 0x02f4e6 to 0x022917 (0xcbce bytes) 000974 1727764533.239555 mapwrite 0x01f0e3 thru 0x02739c (0x82ba bytes) 000977 1727764533.352682 mapwrite 0x013643 thru 0x01ea09 (0xb3c7 bytes) 000978 1727764533.447972 mapwrite 0x017383 thru 0x01cb19 (0x5797 bytes) 000979 1727764533.505916 trunc from 0x02739d to 0x03b5c5 (0x14229 bytes) 000980 1727764533.521491 write 0x036745 thru 0x03ffff (0x98bb bytes) 000981 1727764533.528045 trunc from 0x040000 to 0x0277d2 (0x1882d bytes) 000982 1727764533.541556 read 0x01dd9b thru 0x0277d2 (0x9a38 bytes) 000983 1727764533.542083 write 0x00462f thru 0x00672b (0x20fd bytes) 000985 1727764533.543135 mapwrite 0x01d856 thru 0x028fa5 (0xb750 bytes) 000986 1727764533.626185 write 0x02fd7d thru 0x037a29 (0x7cad bytes) 000987 1727764533.628950 write 0x00ea07 thru 0x01dba1 (0xf19b bytes) 000988 1727764533.635115 mapread 0x01f328 thru 0x0219cd (0x26a6 bytes) 000989 1727764533.636427 mapwrite 0x0172ed thru 0x0254d5 (0xe1e9 bytes) 000990 1727764533.702701 mapread 0x001991 thru 0x0041b5 (0x2825 bytes) 000991 1727764533.703832 mapread 0x034f32 thru 0x037a29 (0x2af8 bytes) 000992 1727764533.709542 MIRROR_EXTEND to 4 mirrors 000993 1727764534.205845 write 0x01b0b2 thru 0x028800 (0xd74f bytes) 000995 1727764534.276408 mapread 0x01c88b thru 0x01e6dc (0x1e52 bytes) 000996 1727764534.277386 read 0x01dce8 thru 0x01e2a8 (0x5c1 bytes) 000998 1727764534.283745 MIRROR_EXTEND to 5 mirrors 001000 1727764534.671431 mapread 0x019764 thru 0x0219d0 (0x826d bytes) All operations completed A-OK! PASS 70a (45s) == sanity-flr test 200a: stress test ===================== 02:35:37 (1727764537) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7702144 .. Extending file size to 3209056 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3211872 .. Extending file size to 5141024 .. Extending file size to 9296288 .. Extending file size to 6861696 .. Extending file size to 7323584 .. Extending file size to 8402624 .. Extending file size to 8751360 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 26063 26064 26065 26069 26071 failed 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) 192.168.203.103@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (72s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:36:50 (1727764610) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7641632 .. Extending file size to 7222784 .. done Extending file size to 7444800 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9066560 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8170784 .. Extending file size to 8560704 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8633216 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8786912 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9130752 .. Extending file size to 9206912 .. Waiting 27746 27748 27749 27750 27751 done 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) 192.168.203.103@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (69s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:37:59 (1727764679) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0495835 s, 42.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.272961 s, 7.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:38:07 (1727764687) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0458897 s, 45.7 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 02:38:11 (1727764691) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c6:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x40e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x40e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x40f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x429:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:38:14 (1727764694) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0643231 s, 32.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x410:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:38:18 (1727764698) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0989809 s, 21.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x411:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x412:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x411:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x412:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42d:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:38:22 (1727764702) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0780904 s, 26.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x413:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x413:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x42f:0x0] } PASS 204c (26s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:38:48 (1727764728) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0551005 s, 19.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x414:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x431:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x430:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x414:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x431:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x430:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x415:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:39:34 (1727764774) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x432:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x433:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0595349 s, 35.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x432:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x433:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x416:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x432:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x433:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x416:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:40:21 (1727764821) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. Creating new pool oleg303-server: Pool lustre.test_204f created Adding targets to pool oleg303-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg303-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0524133 s, 40.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x434:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x417:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x434:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x417:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg303-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg303-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg303-server: Pool lustre.test_204f destroyed PASS 204f (56s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:41:17 (1727764877) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:41:19 (1727764879) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:41:22 (1727764882) oleg303-server: Pool lustre.test_206 created Adding targets to pool oleg303-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg303-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg303-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg303-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg303-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:41:37 (1727764897) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0456153 s, 23.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215893 s, 48.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x4d0d6f4a Mirror 3: 0x4d0d6f4a PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:41:41 (1727764901) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:41:43 (1727764903) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:41:44 (1727764904) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0741265 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1b50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x740000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x740000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x740000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x13f0000 Read this (4 bytes): BBBB 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) PASS 209a (19s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:42:03 (1727764923) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00891761 s, 919 kB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:42:06 (1727764926) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0214437 s, 48.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x3eb lmm_fid: [0x200000404:0x3eb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x3eb lmm_fid: [0x200000404:0x3eb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x421:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834923 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-10-01 02:42:07.000000000 -0400 Modify: 2024-10-01 02:42:07.000000000 -0400 Change: 2024-10-01 02:42:08.000000000 -0400 Birth: - PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:42:10 (1727764930) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test complete, duration 925 sec ============ 02:42:11 (1727764931) === sanity-flr: start cleanup 02:42:11 (1727764931) === === sanity-flr: finish cleanup 02:42:13 (1727764933) ===