-----============= acceptance-small: sanity-flr ============----- Mon Apr 15 15:10:38 EDT 2024 excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:10:42 (1713208242) === oleg239-client.virtnet: executing check_config_client /mnt/lustre oleg239-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg239-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a589800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a589800.idle_timeout=debug disable quota as required oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:10:48 (1713208248) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6913] debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0a: lfs mirror create with -N option == 15:10:49 (1713208249) PASS 0a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:10:54 (1713208254) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:10:56 (1713208256) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0d: lfs mirror extend with -N option == 15:10:59 (1713208259) PASS 0d (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:11:04 (1713208264) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:11:07 (1713208267) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0g: lfs mirror create flags support === 15:11:09 (1713208269) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012a589800.stats=clear osc.lustre-OST0001-osc-ffff88012a589800.stats=clear PASS 0g (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:11:12 (1713208272) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:11:16 (1713208276) PASS 0j (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:11:19 (1713208279) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 1: create components with setstripe options ========================================================== 15:11:22 (1713208282) lfs mirror mirror: cannot get UNLOCK lease, ext 4: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 2: create components from existing files ========================================================== 15:11:25 (1713208285) PASS 2 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:11:28 (1713208288) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:11:30 (1713208290) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:11:33 (1713208293) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 7: Create mirror with DoM component === 15:11:37 (1713208297) PASS 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:11:41 (1713208301) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.329816 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0483972 s, 21.7 MB/s PASS 21 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:11:45 (1713208305) 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0998923 s, 168 MB/s PASS 22 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:11:48 (1713208308) Stopping /mnt/lustre-ost1 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a589800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a589800.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg239-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a589800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a589800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg239-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:12:09 (1713208329) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0555105 s, 75.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a589800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a589800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg239-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (7s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:12:18 (1713208338) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a589800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a589800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg239-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a589800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a589800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg239-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:12:34 (1713208354) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.744979 s, 70.4 MB/s -rw-r--r-- 1 root root 52428800 Apr 15 15:12 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.155s user 0m0.000s sys 0m0.105s Stopping /mnt/lustre-ost1 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a589800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a589800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.173s user 0m0.001s sys 0m0.158s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg239-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a589800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a589800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.185s user 0m0.000s sys 0m0.156s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg239-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg239-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a589800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a589800.ost_server_uuid in FULL state after 0 sec PASS 33b (20s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:12:55 (1713208375) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:12:58 (1713208378) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:13:00 (1713208380) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 35: allow to write to mirrored files == 15:13:02 (1713208382) PASS 35 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36a: write to mirrored files ========== 15:13:05 (1713208385) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.131513 s, 95.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:13:10 (1713208390) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00662766 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00722853 s, 145 MB/s PASS 36b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:13:12 (1713208392) PASS 36c (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:13:16 (1713208396) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000572299 s, 1.8 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000430657 s, 2.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00063447 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000325194 s, 3.1 MB/s flv=10 olv=10 PASS 36d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 37: mirror I/O API verification ======= 15:13:20 (1713208400) create test files with size 15752660 .. 7d7a2d23dc47aa2487f59ffd322ad538 - 653615a1cd616b0cd19a602e8aa165c8 - 4b48b981bf9904ce0875e0602db36ea4 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg239-client.virtnet /mnt/lustre (opts:) Starting client: oleg239-client.virtnet: -o user_xattr,flock oleg239-server@tcp:/lustre /mnt/lustre PASS 37 (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 38: resync ============================ 15:13:25 (1713208405) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:13:30 (1713208410) PASS 39 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:13:33 (1713208413) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206021 s, 50.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4c:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00668089 s, 157 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 component0: lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 component1: lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 component2: lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component3: lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 component4: lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component5: lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 component6: lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 41: lfs mirror resync check =========== 15:13:36 (1713208416) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0237038 s, 88.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.078523 s, 53.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0405515 s, 51.7 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 42: lfs mirror verify ================= 15:13:40 (1713208420) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43a: mirror pick on write ============= 15:13:42 (1713208422) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:13:44 (1713208424) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00636108 s, 165 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44a: lfs mirror split check =========== 15:13:47 (1713208427) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:13:49 (1713208429) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x55:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:13:53 (1713208433) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0568422 s, 184 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg239-client.virtnet /mnt/lustre (opts:) Starting client: oleg239-client.virtnet: -o user_xattr,flock oleg239-server@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:13:56 (1713208436) llite.lustre-ffff88012a4f5000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0619686 s, 169 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg239-client.virtnet /mnt/lustre (opts:) Starting client: oleg239-client.virtnet: -o user_xattr,flock oleg239-server@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff88012dba7800.xattr_cache=1 PASS 44e (3s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:14:01 (1713208441) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 46: Verify setstripe --copy option ==== 15:14:09 (1713208449) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 47: Verify mirror obj alloc =========== 15:14:16 (1713208456) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 48: Verify snapshot mirror ============ 15:14:19 (1713208459) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0688994 s, 45.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.055043 s, 57.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 12159dd0a267bc27e4ca8b7c739e96dc - mirror 1 checksum: 46daf2b7b1e71f19989d8fc002fc0cca - mirror 2 checksum: 12159dd0a267bc27e4ca8b7c739e96dc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Apr 15 15:14:19 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Apr 15 15:14:19 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 12159dd0a267bc27e4ca8b7c739e96dc - mirror 1 checksum: 46daf2b7b1e71f19989d8fc002fc0cca - mirror 2 checksum: 46daf2b7b1e71f19989d8fc002fc0cca - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Apr 15 15:14:19 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Apr 15 15:14:19 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } PASS 48 (2s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:14:23 (1713208463) Failing mds1 on oleg239-server Stopping /mnt/lustre-mds1 (opts:) on oleg239-server 15:14:25 (1713208465) shut down Failover mds1 to oleg239-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-MDT0000 15:14:38 (1713208478) targets are mounted 15:14:38 (1713208478) facet_failover done oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00756037 s, 139 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x14 lmm_fid: [0x200000404:0x14:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 50A (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:14:44 (1713208484) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50b: mirror rsync handles sparseness == 15:14:46 (1713208486) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:14:49 (1713208489) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00226465 s, 7.2 MB/s PASS 50c (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 50d: mirror rsync keep holes ========== 15:14:52 (1713208492) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:14:54 (1713208494) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:14:56 (1713208496) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) PASS 60b (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:15:01 (1713208501) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:15:20 (1713208520) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:15:39 (1713208539) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (16s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:15:57 (1713208557) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70: mirror create and split race ====== 15:16:00 (1713208560) mirror create pid 26599 mirror split pid 26600 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26599 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4697: 26600 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 70a: flr mode fsx test ================ 15:17:03 (1713208623) Seed set to 8623 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x2730b 000001 1713208623.739331 trunc from 00000000 to 0x02730a (0x2730b bytes) 000007 1713208623.744438 mapread 0x01ce46 thru 0x02730a (0xa4c5 bytes) 000009 1713208623.751286 trunc from 0x02730b to 0x01ae04 (0xc506 bytes) 000011 1713208623.755917 MIRROR_EXTEND to 2 mirrors 000013 1713208623.862686 read 0x01195e thru 0x015ab1 (0x4154 bytes) 000015 1713208623.866399 trunc from 0x01ae05 to 0x026e67 (0xc063 bytes) 000016 1713208623.884088 mapwrite 0x01d950 thru 0x029a88 (0xc139 bytes) 000018 1713208623.911593 write 0x016fab thru 0x020faa (0xa000 bytes) 000019 1713208623.916653 MIRROR_EXTEND to 3 mirrors 000026 1713208624.040620 mapwrite 0x03793c thru 0x039844 (0x1f09 bytes) 000028 1713208624.077633 write 0x01936e thru 0x0284ab (0xf13e bytes) 000029 1713208624.082475 MIRROR_RESYNC 3 mirrors 000030 1713208624.194324 mapread 0x018776 thru 0x025181 (0xca0c bytes) 000033 1713208624.200071 read 0x035feb thru 0x039844 (0x385a bytes) 000036 1713208624.201572 read 0x013afd thru 0x01d914 (0x9e18 bytes) 000037 1713208624.201842 write 0x003f08 thru 0x00ccac (0x8da5 bytes) 000038 1713208624.218112 mapread 0x03716b thru 0x039844 (0x26da bytes) 000040 1713208624.218961 trunc from 0x039845 to 0x00f261 (0x2a5e3 bytes) 000041 1713208624.228362 MIRROR_EXTEND to 4 mirrors truncating to largest ever: 0x3044f 000042 1713208624.327762 trunc from 0x00f262 to 0x03044e (0x211ed bytes) 000044 1713208624.345827 read 0x00a035 thru 0x011660 (0x762c bytes) 000046 1713208624.348332 write 0x018bc7 thru 0x0240d7 (0xb511 bytes) 000047 1713208624.352162 MIRROR_EXTEND to 5 mirrors 000048 1713208624.466109 mapwrite 0x014633 thru 0x0159a4 (0x1372 bytes) 000049 1713208624.510406 mapread 0x02b99c thru 0x03044e (0x4ab3 bytes) 000050 1713208624.514485 MIRROR_EXTEND to 6 mirrors 000051 1713208624.617179 MIRROR_EXTEND to 7 mirrors 000053 1713208624.723818 trunc from 0x03044f to 0x026617 (0x9e37 bytes) 000058 1713208624.742649 trunc from 0x026618 to 0x005494 (0x21183 bytes) 000059 1713208624.746995 mapread 0x001e3a thru 0x004677 (0x283e bytes) 000060 1713208624.751438 mapread 0x00469b thru 0x005494 (0xdfa bytes) 000061 1713208624.751869 mapread 0x0020b8 thru 0x005494 (0x33dd bytes) 000062 1713208624.752288 mapwrite 0x030263 thru 0x03d3fc (0xd19a bytes) 000063 1713208624.774843 read 0x0315cf thru 0x03cf21 (0xb953 bytes) 000066 1713208624.777033 MIRROR_RESYNC 7 mirrors 000067 1713208624.974773 mapread 0x00f0eb thru 0x014f2f (0x5e45 bytes) 000068 1713208624.979757 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000069 1713208625.040636 mapread 0x038d38 thru 0x039c2e (0xef7 bytes) 000071 1713208625.044953 MIRROR_EXTEND to 7 mirrors 000072 1713208625.164576 MIRROR_EXTEND to 8 mirrors 000073 1713208625.297498 MIRROR_EXTEND to 9 mirrors 000074 1713208625.438167 mapread 0x032551 thru 0x0338c8 (0x1378 bytes) 000075 1713208625.443948 read 0x009508 thru 0x017d34 (0xe82d bytes) 000076 1713208625.448723 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000078 1713208625.561041 MIRROR_EXTEND to 9 mirrors 000079 1713208625.716575 mapwrite 0x010989 thru 0x012993 (0x200b bytes) 000080 1713208625.763645 mapwrite 0x03c4ba thru 0x03ffff (0x3b46 bytes) 000082 1713208625.788024 mapread 0x036170 thru 0x03ffff (0x9e90 bytes) 000083 1713208625.788785 mapread 0x01711e thru 0x01980c (0x26ef bytes) 000084 1713208625.789414 read 0x0360d9 thru 0x0369de (0x906 bytes) 000085 1713208625.789875 write 0x024145 thru 0x02aeb2 (0x6d6e bytes) 000087 1713208625.792668 write 0x021756 thru 0x024b44 (0x33ef bytes) 000092 1713208625.794147 mapwrite 0x032b37 thru 0x035070 (0x253a bytes) 000093 1713208625.810040 mapwrite 0x027f0b thru 0x029538 (0x162e bytes) 000094 1713208625.822969 write 0x007be1 thru 0x01124a (0x966a bytes) 000096 1713208625.826170 trunc from 0x040000 to 0x024baa (0x1b455 bytes) 000097 1713208625.830187 mapwrite 0x009384 thru 0x011f22 (0x8b9f bytes) 000098 1713208625.845349 write 0x017f81 thru 0x0216c6 (0x9746 bytes) 000099 1713208625.846184 trunc from 0x024bab to 0x00dc32 (0x16f78 bytes) 000100 1713208625.850414 read 0x0064e1 thru 0x00b424 (0x4f44 bytes) 000101 1713208625.850896 mapread 0x0076f9 thru 0x00db79 (0x6481 bytes) 000103 1713208625.851714 trunc from 0x00dc33 to 0x0175ed (0x99bb bytes) 000104 1713208625.856261 trunc from 0x0175ee to 0x024f04 (0xd917 bytes) 000106 1713208625.861312 mapread 0x01d544 thru 0x021e8d (0x494a bytes) 000108 1713208625.865590 write 0x00636d thru 0x00da16 (0x76aa bytes) 000110 1713208625.866597 mapwrite 0x00d289 thru 0x00e582 (0x12fa bytes) 000111 1713208625.884923 MIRROR_RESYNC 9 mirrors 000113 1713208626.145468 mapread 0x01fe2c thru 0x024f04 (0x50d9 bytes) 000114 1713208626.151249 MIRROR_RESYNC 9 mirrors 000117 1713208626.204553 read 0x009ccc thru 0x015ed9 (0xc20e bytes) 000121 1713208626.208622 MIRROR_EXTEND to 10 mirrors 000122 1713208626.349117 write 0x031dd1 thru 0x03c184 (0xa3b4 bytes) 000123 1713208626.365041 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000124 1713208626.414047 mapwrite 0x001b1f thru 0x010f49 (0xf42b bytes) 000126 1713208626.439947 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000129 1713208626.488253 mapwrite 0x0231d3 thru 0x030507 (0xd335 bytes) 000131 1713208626.509697 write 0x00950e thru 0x00f8e5 (0x63d8 bytes) 000132 1713208626.512574 mapwrite 0x0342d6 thru 0x03ffff (0xbd2a bytes) 000134 1713208626.533220 write 0x02d3f7 thru 0x03b679 (0xe283 bytes) truncating to largest ever: 0x31b20 000136 1713208626.534407 trunc from 0x040000 to 0x031b1f (0xe4e0 bytes) 000139 1713208626.538827 trunc from 0x031b20 to 0x02ef2d (0x2bf2 bytes) 000140 1713208626.543802 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000141 1713208626.590848 write 0x00a094 thru 0x019c56 (0xfbc3 bytes) 000143 1713208626.595503 MIRROR_RESYNC 7 mirrors 000144 1713208626.815649 mapwrite 0x03eb99 thru 0x03ffff (0x1467 bytes) 000145 1713208626.853119 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000147 1713208626.906428 mapwrite 0x00b83e thru 0x00c45a (0xc1d bytes) 000148 1713208626.924364 MIRROR_EXTEND to 7 mirrors 000149 1713208627.046338 write 0x01c993 thru 0x0224c9 (0x5b37 bytes) 000151 1713208627.063605 write 0x016b47 thru 0x024a85 (0xdf3f bytes) 000153 1713208627.068151 mapread 0x03e670 thru 0x03ffff (0x1990 bytes) 000154 1713208627.070408 read 0x037bf0 thru 0x03ffff (0x8410 bytes) 000155 1713208627.071838 write 0x03738f thru 0x038c16 (0x1888 bytes) 000156 1713208627.072316 mapwrite 0x01171d thru 0x014c55 (0x3539 bytes) 000158 1713208627.089076 MIRROR_RESYNC 7 mirrors 000160 1713208627.268671 mapread 0x01c74f thru 0x020347 (0x3bf9 bytes) 000163 1713208627.273341 read 0x02b526 thru 0x034167 (0x8c42 bytes) 000165 1713208627.275621 mapwrite 0x017d73 thru 0x0207f0 (0x8a7e bytes) 000166 1713208627.320008 read 0x011d08 thru 0x01f11b (0xd414 bytes) 000167 1713208627.320508 read 0x039501 thru 0x03c1d3 (0x2cd3 bytes) 000168 1713208627.324471 MIRROR_EXTEND to 8 mirrors 000170 1713208627.439541 mapwrite 0x0079b7 thru 0x017091 (0xf6db bytes) 000172 1713208627.494692 MIRROR_RESYNC 8 mirrors 000173 1713208627.703703 write 0x02048f thru 0x02d0d3 (0xcc45 bytes) 000175 1713208627.721816 read 0x01659f thru 0x01d06d (0x6acf bytes) 000177 1713208627.724380 read 0x012522 thru 0x0167f7 (0x42d6 bytes) truncating to largest ever: 0x3f180 000179 1713208627.727555 trunc from 0x040000 to 0x03f17f (0xe80 bytes) 000180 1713208627.735833 read 0x038b63 thru 0x03ec48 (0x60e6 bytes) 000181 1713208627.736322 read 0x03d4b6 thru 0x03f17f (0x1cca bytes) 000183 1713208627.736993 mapread 0x000a9a thru 0x00ff8f (0xf4f6 bytes) 000184 1713208627.737506 write 0x01ccba thru 0x026b51 (0x9e98 bytes) 000186 1713208627.738418 write 0x02e3f5 thru 0x039d12 (0xb91e bytes) 000191 1713208627.740049 read 0x014f65 thru 0x01b274 (0x6310 bytes) 000192 1713208627.740329 trunc from 0x03f180 to 0x001a40 (0x3d73f bytes) 000197 1713208627.745231 read 0x000b36 thru 0x001a40 (0xf0b bytes) 000198 1713208627.745602 mapread 0x0017e7 thru 0x001a40 (0x25a bytes) 000203 1713208627.747365 mapwrite 0x0274f1 thru 0x034f98 (0xdaa8 bytes) 000204 1713208627.773053 mapwrite 0x0364b4 thru 0x03ffff (0x9b4c bytes) 000205 1713208627.792877 trunc from 0x040000 to 0x011898 (0x2e767 bytes) 000212 1713208627.800044 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000216 1713208627.846498 write 0x002047 thru 0x0072aa (0x5264 bytes) 000219 1713208627.851152 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000220 1713208627.900054 mapwrite 0x012a25 thru 0x01435c (0x1938 bytes) 000224 1713208627.924064 write 0x02b6f9 thru 0x02fef4 (0x47fc bytes) 000225 1713208627.924800 trunc from 0x02fef5 to 0x00e56a (0x2198a bytes) 000228 1713208627.928915 write 0x005178 thru 0x005c2c (0xab5 bytes) 000229 1713208627.931213 mapwrite 0x034410 thru 0x03ced0 (0x8ac1 bytes) 000233 1713208627.950054 read 0x0173e5 thru 0x018398 (0xfb4 bytes) 000234 1713208627.951649 mapread 0x0241a2 thru 0x02636e (0x21cd bytes) 000235 1713208627.953973 trunc from 0x03ced1 to 0x00c9b3 (0x3051d bytes) 000237 1713208627.959225 MIRROR_RESYNC 6 mirrors 000239 1713208628.130913 MIRROR_RESYNC 6 mirrors 000240 1713208628.170017 mapwrite 0x033e2b thru 0x03fdeb (0xbfc1 bytes) 000244 1713208628.207657 mapread 0x03204c thru 0x03f945 (0xd8fa bytes) 000245 1713208628.208294 mapread 0x0217e6 thru 0x021848 (0x63 bytes) 000249 1713208628.211777 mapread 0x007937 thru 0x0164f7 (0xebc1 bytes) 000254 1713208628.217141 mapread 0x01018a thru 0x012f83 (0x2dfa bytes) 000255 1713208628.218907 MIRROR_RESYNC 6 mirrors 000256 1713208628.393458 trunc from 0x03fdec to 0x010fba (0x2ee31 bytes) 000257 1713208628.414203 MIRROR_EXTEND to 7 mirrors 000258 1713208628.517283 mapwrite 0x02dea0 thru 0x036641 (0x87a2 bytes) 000259 1713208628.557185 write 0x01508a thru 0x01f67e (0xa5f5 bytes) 000260 1713208628.561165 read 0x021b13 thru 0x02dec7 (0xc3b5 bytes) 000263 1713208628.564095 mapread 0x02f7ef thru 0x032dbd (0x35cf bytes) 000266 1713208628.565390 trunc from 0x036642 to 0x03b4d8 (0x4e97 bytes) 000268 1713208628.570508 mapwrite 0x0382e8 thru 0x03cae2 (0x47fb bytes) 000269 1713208628.592340 MIRROR_EXTEND to 8 mirrors 000270 1713208628.697183 mapread 0x034167 thru 0x03b02c (0x6ec6 bytes) 000271 1713208628.703854 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000273 1713208628.765058 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000274 1713208628.822967 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000275 1713208628.881374 mapwrite 0x0090b9 thru 0x00a76c (0x16b4 bytes) 000276 1713208628.927233 MIRROR_RESYNC 5 mirrors 000280 1713208629.080348 trunc from 0x03cae3 to 0x02c726 (0x103bc bytes) 000281 1713208629.094694 read 0x003ce0 thru 0x00667a (0x299b bytes) 000282 1713208629.096298 trunc from 0x02c727 to 0x00a62c (0x220fa bytes) 000283 1713208629.100279 read 0x003903 thru 0x007f2e (0x462c bytes) 000286 1713208629.102671 write 0x022b99 thru 0x0256fa (0x2b62 bytes) 000287 1713208629.103187 mapwrite 0x03b77f thru 0x03ffff (0x4881 bytes) 000288 1713208629.124462 write 0x0083c9 thru 0x014a57 (0xc68f bytes) 000289 1713208629.125485 trunc from 0x040000 to 0x005f54 (0x3a0ab bytes) 000291 1713208629.129414 write 0x015550 thru 0x024ce9 (0xf79a bytes) 000293 1713208629.130527 read 0x017a96 thru 0x01830e (0x879 bytes) 000297 1713208629.131472 mapread 0x01aec7 thru 0x024ce9 (0x9e23 bytes) 000299 1713208629.132181 write 0x013cce thru 0x0237ff (0xfb32 bytes) 000300 1713208629.134653 mapwrite 0x02287a thru 0x02b902 (0x9089 bytes) 000301 1713208629.154186 mapread 0x004de6 thru 0x01387b (0xea96 bytes) 000302 1713208629.154957 write 0x01c180 thru 0x020359 (0x41da bytes) 000303 1713208629.156981 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000304 1713208629.201356 trunc from 0x02b903 to 0x013b80 (0x17d82 bytes) 000306 1713208629.205957 MIRROR_EXTEND to 5 mirrors 000307 1713208629.320362 mapread 0x00d078 thru 0x00d31c (0x2a5 bytes) 000308 1713208629.323678 write 0x00f6df thru 0x00fde0 (0x702 bytes) 000310 1713208629.338718 mapread 0x00a835 thru 0x01139e (0x6b6a bytes) 000311 1713208629.341190 mapwrite 0x00377a thru 0x010fb9 (0xd840 bytes) 000313 1713208629.359422 mapwrite 0x003e42 thru 0x006bac (0x2d6b bytes) 000314 1713208629.371005 trunc from 0x013b81 to 0x01d9bc (0x9e3c bytes) 000316 1713208629.380058 MIRROR_EXTEND to 6 mirrors 000318 1713208629.476467 write 0x03d571 thru 0x03ffff (0x2a8f bytes) 000321 1713208629.492013 mapread 0x030fe4 thru 0x03fefb (0xef18 bytes) 000322 1713208629.498229 trunc from 0x040000 to 0x037566 (0x8a99 bytes) 000324 1713208629.506969 write 0x03fd98 thru 0x03ffff (0x268 bytes) 000325 1713208629.507428 read 0x029a26 thru 0x02bdb0 (0x238b bytes) 000326 1713208629.510231 MIRROR_EXTEND to 7 mirrors 000330 1713208629.619544 mapread 0x006f14 thru 0x00e1df (0x72cc bytes) 000331 1713208629.625596 mapwrite 0x009c9a thru 0x00b88f (0x1bf6 bytes) 000334 1713208629.669768 write 0x02c468 thru 0x030a80 (0x4619 bytes) 000336 1713208629.673745 MIRROR_EXTEND to 8 mirrors 000337 1713208629.786138 read 0x00c5ce thru 0x0152b4 (0x8ce7 bytes) 000339 1713208629.790097 mapwrite 0x004acf thru 0x00f02b (0xa55d bytes) 000340 1713208629.837286 trunc from 0x040000 to 0x01e05f (0x21fa0 bytes) 000341 1713208629.844943 read 0x00c2c2 thru 0x01161c (0x535b bytes) 000342 1713208629.847304 mapwrite 0x023ec5 thru 0x02e508 (0xa644 bytes) 000344 1713208629.870298 MIRROR_RESYNC 8 mirrors 000345 1713208630.098711 read 0x01fca3 thru 0x02b2cf (0xb62d bytes) 000346 1713208630.100916 trunc from 0x02e509 to 0x00d48d (0x2107b bytes) 000347 1713208630.118659 write 0x03532c thru 0x03ad96 (0x5a6b bytes) 000349 1713208630.119590 mapwrite 0x02acbd thru 0x02e618 (0x395c bytes) 000352 1713208630.140326 mapread 0x018d43 thru 0x022cf0 (0x9fae bytes) 000353 1713208630.144905 mapread 0x000221 thru 0x001b56 (0x1936 bytes) 000355 1713208630.148659 mapread 0x00c9e8 thru 0x0179c0 (0xafd9 bytes) 000356 1713208630.150398 mapwrite 0x0374d6 thru 0x03ffff (0x8b2a bytes) 000359 1713208630.170996 mapread 0x0374ab thru 0x03fe26 (0x897c bytes) 000360 1713208630.171633 write 0x0309ef thru 0x0326ca (0x1cdc bytes) 000361 1713208630.172407 write 0x026c8f thru 0x02d8f2 (0x6c64 bytes) 000362 1713208630.173278 trunc from 0x040000 to 0x036e91 (0x916e bytes) 000366 1713208630.178382 read 0x00a146 thru 0x01564d (0xb508 bytes) 000369 1713208630.179085 mapread 0x028885 thru 0x034b71 (0xc2ed bytes) 000370 1713208630.180759 MIRROR_RESYNC 8 mirrors 000371 1713208630.404650 MIRROR_EXTEND to 9 mirrors 000372 1713208630.554551 write 0x02e089 thru 0x03a368 (0xc2e0 bytes) 000374 1713208630.573030 read 0x013b0e thru 0x014419 (0x90c bytes) 000377 1713208630.575497 mapread 0x0142e7 thru 0x01a57b (0x6295 bytes) 000380 1713208630.579725 trunc from 0x03a369 to 0x0025b7 (0x37db1 bytes) 000381 1713208630.590064 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000382 1713208630.643231 MIRROR_RESYNC 8 mirrors 000383 1713208630.854586 write 0x015e2f thru 0x01de83 (0x8055 bytes) 000385 1713208630.871422 trunc from 0x01de84 to 0x02ff6d (0x120ea bytes) 000387 1713208630.880596 write 0x01a736 thru 0x027d0a (0xd5d5 bytes) 000388 1713208630.884297 MIRROR_EXTEND to 9 mirrors 000389 1713208630.992061 write 0x002c02 thru 0x004ae2 (0x1ee1 bytes) 000391 1713208631.010770 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000393 1713208631.057398 mapread 0x010b5f thru 0x0158be (0x4d60 bytes) 000394 1713208631.061972 write 0x00e0be thru 0x019056 (0xaf99 bytes) 000395 1713208631.062697 trunc from 0x02ff6e to 0x0040e7 (0x2be86 bytes) 000396 1713208631.070159 mapread 0x003893 thru 0x0040e7 (0x855 bytes) 000398 1713208631.072436 mapwrite 0x003ab9 thru 0x010390 (0xc8d8 bytes) 000399 1713208631.095168 mapread 0x00cde5 thru 0x010390 (0x35ac bytes) 000401 1713208631.098137 MIRROR_EXTEND to 9 mirrors 000403 1713208631.207104 read 0x009361 thru 0x00fdd7 (0x6a77 bytes) 000406 1713208631.214123 mapread 0x008dd5 thru 0x00eb9c (0x5dc8 bytes) 000409 1713208631.221550 mapread 0x00a6e6 thru 0x00e285 (0x3ba0 bytes) 000410 1713208631.224749 trunc from 0x010391 to 0x01b1fc (0xae6c bytes) 000411 1713208631.241296 mapread 0x00ba2e thru 0x01783c (0xbe0f bytes) 000413 1713208631.247073 mapwrite 0x03a7e9 thru 0x03ffff (0x5817 bytes) 000414 1713208631.270082 MIRROR_RESYNC 9 mirrors 000415 1713208631.503466 write 0x03e696 thru 0x03ffff (0x196a bytes) 000417 1713208631.517750 trunc from 0x040000 to 0x020a22 (0x1f5dd bytes) 000418 1713208631.526349 trunc from 0x020a23 to 0x01cf04 (0x3b1e bytes) 000419 1713208631.529780 read 0x00f169 thru 0x017fa0 (0x8e38 bytes) 000421 1713208631.532585 read 0x0117c6 thru 0x01cf04 (0xb73f bytes) 000422 1713208631.536266 MIRROR_EXTEND to 10 mirrors 000423 1713208631.633649 read 0x01cb0d thru 0x01cf04 (0x3f8 bytes) 000427 1713208631.637512 trunc from 0x01cf05 to 0x0377fe (0x1a8fa bytes) 000428 1713208631.657520 mapwrite 0x008b3b thru 0x01634d (0xd813 bytes) 000430 1713208631.680065 mapread 0x01c739 thru 0x01c8a9 (0x171 bytes) 000431 1713208631.682711 mapread 0x0117df thru 0x017506 (0x5d28 bytes) 000433 1713208631.683453 trunc from 0x0377ff to 0x02d8ab (0x9f53 bytes) 000435 1713208631.688212 mapwrite 0x010f5e thru 0x01828e (0x7331 bytes) 000437 1713208631.702598 mapwrite 0x01827a thru 0x01f090 (0x6e17 bytes) 000438 1713208631.716530 trunc from 0x02d8ac to 0x02ad60 (0x2b4b bytes) 000440 1713208631.720272 write 0x029b66 thru 0x0301f0 (0x668b bytes) 000443 1713208631.723273 write 0x0055d2 thru 0x00d6ca (0x80f9 bytes) 000445 1713208631.724125 mapread 0x021bd3 thru 0x02e0b7 (0xc4e5 bytes) 000446 1713208631.724561 mapread 0x00e9bf thru 0x012e31 (0x4473 bytes) 000447 1713208631.724934 trunc from 0x0301f1 to 0x030f8d (0xd9d bytes) 000451 1713208631.729759 trunc from 0x030f8e to 0x02b09a (0x5ef3 bytes) 000452 1713208631.733573 read 0x028860 thru 0x02b09a (0x283b bytes) 000453 1713208631.733882 write 0x00fc32 thru 0x01cc74 (0xd043 bytes) 000455 1713208631.734777 read 0x011ca9 thru 0x01b0a6 (0x93fe bytes) 000456 1713208631.735036 read 0x01633a thru 0x01df0f (0x7bd6 bytes) 000457 1713208631.735424 mapread 0x0111ff thru 0x01b36a (0xa16c bytes) 000459 1713208631.736214 mapwrite 0x005166 thru 0x01103e (0xbed9 bytes) 000461 1713208631.754093 MIRROR_EXTEND to 11 mirrors 000462 1713208631.858227 read 0x009bcf thru 0x018456 (0xe888 bytes) 000464 1713208631.862334 write 0x023937 thru 0x027c6e (0x4338 bytes) 000465 1713208631.879105 mapread 0x018b4a thru 0x028afc (0xffb3 bytes) 000467 1713208631.884621 read 0x01012c thru 0x016322 (0x61f7 bytes) 000469 1713208631.887065 trunc from 0x02b09b to 0x027bd7 (0x34c3 bytes) 000471 1713208631.895124 read 0x013e66 thru 0x016cc9 (0x2e64 bytes) 000473 1713208631.895677 mapread 0x021935 thru 0x025ecf (0x459b bytes) 000475 1713208631.896387 read 0x025d65 thru 0x027bd7 (0x1e73 bytes) 000476 1713208631.897791 MIRROR_EXTEND to 12 mirrors 000478 1713208632.003322 write 0x01f9a3 thru 0x0289a7 (0x9005 bytes) 000479 1713208632.019944 MIRROR_EXTEND to 13 mirrors 000480 1713208632.151783 MIRROR_RESYNC 13 mirrors 000481 1713208632.457317 mapread 0x01e099 thru 0x023390 (0x52f8 bytes) 000484 1713208632.461610 mapwrite 0x010ad2 thru 0x019b19 (0x9048 bytes) 000486 1713208632.508933 read 0x00681c thru 0x015ef5 (0xf6da bytes) 000487 1713208632.510628 trunc from 0x0289a8 to 0x03c2ae (0x13907 bytes) 000489 1713208632.515276 write 0x005317 thru 0x00984f (0x4539 bytes) 000491 1713208632.517888 read 0x02116b thru 0x02c662 (0xb4f8 bytes) 000492 1713208632.520476 mapwrite 0x03590e thru 0x03d37a (0x7a6d bytes) 000493 1713208632.539794 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000499 1713208632.585863 read 0x03bdc8 thru 0x03d37a (0x15b3 bytes) 000503 1713208632.588191 mapread 0x009b05 thru 0x01177f (0x7c7b bytes) 000504 1713208632.592365 mapwrite 0x00682d thru 0x014b11 (0xe2e5 bytes) 000505 1713208632.611590 mapwrite 0x03d966 thru 0x03ffff (0x269a bytes) 000509 1713208632.633427 MIRROR_RESYNC 12 mirrors 000510 1713208632.881945 mapread 0x0393a1 thru 0x03ffff (0x6c5f bytes) 000511 1713208632.886260 mapread 0x01701f thru 0x01ba37 (0x4a19 bytes) 000512 1713208632.888323 mapread 0x0254dc thru 0x0319a2 (0xc4c7 bytes) 000513 1713208632.892770 MIRROR_EXTEND to 13 mirrors 000514 1713208633.026745 trunc from 0x040000 to 0x00eaef (0x31510 bytes) 000515 1713208633.046897 trunc from 0x00eaf0 to 0x013718 (0x4c29 bytes) 000516 1713208633.051093 mapwrite 0x00e271 thru 0x018af9 (0xa889 bytes) 000517 1713208633.076007 mapwrite 0x00a22f thru 0x00bb3f (0x1911 bytes) 000521 1713208633.088247 mapwrite 0x03e52c thru 0x03ffff (0x1ad4 bytes) 000523 1713208633.104803 mapread 0x0341e4 thru 0x036758 (0x2575 bytes) 000525 1713208633.107298 write 0x00eb06 thru 0x01ce36 (0xe331 bytes) 000529 1713208633.111253 MIRROR_EXTEND to 14 mirrors 000530 1713208633.253334 mapwrite 0x028f11 thru 0x0380cf (0xf1bf bytes) 000531 1713208633.306792 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000533 1713208633.354348 mapread 0x036602 thru 0x036877 (0x276 bytes) 000538 1713208633.357360 write 0x0051f6 thru 0x008139 (0x2f44 bytes) 000539 1713208633.360292 trunc from 0x040000 to 0x027a83 (0x1857c bytes) 000541 1713208633.364323 mapwrite 0x03f087 thru 0x03ffff (0xf79 bytes) 000542 1713208633.382894 mapwrite 0x0212fc thru 0x022a8c (0x1791 bytes) 000543 1713208633.403490 mapwrite 0x025614 thru 0x02aaae (0x549b bytes) 000544 1713208633.418354 write 0x030468 thru 0x03be92 (0xba2b bytes) 000545 1713208633.419300 mapread 0x016b00 thru 0x024d0b (0xe20c bytes) 000547 1713208633.424232 MIRROR_EXTEND to 14 mirrors 000548 1713208633.552622 MIRROR_RESYNC 14 mirrors 000549 1713208633.841410 MIRROR_RESYNC 14 mirrors 000550 1713208633.914798 MIRROR_RESYNC 14 mirrors 000551 1713208633.985625 trunc from 0x040000 to 0x00ac0a (0x353f5 bytes) 000553 1713208634.002504 read 0x002f3e thru 0x004cdd (0x1da0 bytes) 000555 1713208634.005687 MIRROR_EXTEND to 15 mirrors 000556 1713208634.116547 MIRROR_EXTEND to 16 mirrors 000557 1713208634.224109 MIRROR_SPLIT mirror 9 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000558 1713208634.276265 trunc from 0x00ac0b to 0x02a794 (0x1fb8a bytes) 000559 1713208634.294878 read 0x001f59 thru 0x01166e (0xf716 bytes) 000563 1713208634.298793 trunc from 0x02a795 to 0x024110 (0x6684 bytes) 000564 1713208634.303885 mapread 0x0125b5 thru 0x01f145 (0xcb91 bytes) 000566 1713208634.311288 read 0x00b2df thru 0x010bec (0x590e bytes) 000568 1713208634.312077 write 0x0323a6 thru 0x03f6e8 (0xd343 bytes) 000569 1713208634.312887 mapread 0x00fb03 thru 0x016b40 (0x703e bytes) 000572 1713208634.313819 trunc from 0x03f6e9 to 0x005d4e (0x3999a bytes) 000573 1713208634.317879 mapread 0x0026c2 thru 0x005d4e (0x368d bytes) 000574 1713208634.318299 trunc from 0x005d4f to 0x00e34d (0x85ff bytes) 000575 1713208634.321378 read 0x00d4fd thru 0x00e34d (0xe51 bytes) 000576 1713208634.323354 write 0x03a989 thru 0x03ebbf (0x4237 bytes) 000577 1713208634.323936 mapwrite 0x0151aa thru 0x020797 (0xb5ee bytes) 000578 1713208634.343933 MIRROR_EXTEND to 16 mirrors 000582 1713208634.457350 read 0x01a948 thru 0x029d46 (0xf3ff bytes) 000583 1713208634.461573 MIRROR_RESYNC 16 mirrors 000584 1713208634.856805 mapwrite 0x009b7c thru 0x011169 (0x75ee bytes) 000586 1713208634.905001 read 0x0315b6 thru 0x037292 (0x5cdd bytes) 000587 1713208634.907469 mapread 0x030e0e thru 0x033c90 (0x2e83 bytes) 000588 1713208634.909680 read 0x038132 thru 0x03ebbf (0x6a8e bytes) 000591 1713208634.912468 mapwrite 0x03b10d thru 0x03ffff (0x4ef3 bytes) 000593 1713208634.935038 trunc from 0x040000 to 0x011d8a (0x2e275 bytes) 000596 1713208634.939284 trunc from 0x011d8b to 0x02d3ea (0x1b660 bytes) 000597 1713208634.942894 mapwrite 0x01d43a thru 0x026e56 (0x9a1d bytes) 000599 1713208634.970890 read 0x008dc4 thru 0x00dd4f (0x4f8c bytes) 000601 1713208634.971805 trunc from 0x02d3eb to 0x029f49 (0x34a1 bytes) 000602 1713208634.976546 mapread 0x0289e5 thru 0x029f0c (0x1528 bytes) 000603 1713208634.979257 read 0x01aca4 thru 0x0226ee (0x7a4b bytes) 000607 1713208634.980465 trunc from 0x029f4a to 0x033788 (0x983f bytes) 000609 1713208634.984367 write 0x00bd98 thru 0x0153f3 (0x965c bytes) 000612 1713208634.985612 mapwrite 0x037936 thru 0x0399c4 (0x208f bytes) 000613 1713208635.007086 mapwrite 0x0138f5 thru 0x01c5c8 (0x8cd4 bytes) 000614 1713208635.021271 mapwrite 0x001804 thru 0x00264d (0xe4a bytes) 000616 1713208635.036405 read 0x0264b3 thru 0x02a4e2 (0x4030 bytes) 000617 1713208635.038066 MIRROR_RESYNC 16 mirrors 000620 1713208635.429707 read 0x02545f thru 0x031b09 (0xc6ab bytes) 000621 1713208635.431929 mapwrite 0x038f9d thru 0x0394e9 (0x54d bytes) 000622 1713208635.485927 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000623 1713208635.533998 read 0x014e37 thru 0x0229b5 (0xdb7f bytes) 000624 1713208635.536940 mapwrite 0x0337df thru 0x03fc7d (0xc49f bytes) 000625 1713208635.564749 mapread 0x0057c1 thru 0x007935 (0x2175 bytes) 000626 1713208635.567387 mapwrite 0x03d76a thru 0x03ffff (0x2896 bytes) 000627 1713208635.581521 mapwrite 0x01fcf9 thru 0x020c4d (0xf55 bytes) 000628 1713208635.592425 trunc from 0x040000 to 0x01f7ad (0x20852 bytes) 000629 1713208635.595901 mapread 0x017764 thru 0x01f7ad (0x804a bytes) 000630 1713208635.596364 mapwrite 0x004a71 thru 0x006186 (0x1716 bytes) 000632 1713208635.609651 trunc from 0x01f7ae to 0x0365db (0x16e2e bytes) 000634 1713208635.614244 mapread 0x00476b thru 0x010c49 (0xc4df bytes) 000635 1713208635.614964 write 0x01f7bd thru 0x02a5b7 (0xadfb bytes) 000637 1713208635.617263 trunc from 0x0365dc to 0x00af38 (0x2b6a3 bytes) 000638 1713208635.620724 read 0x0000e8 thru 0x0003b9 (0x2d2 bytes) 000639 1713208635.623389 MIRROR_SPLIT mirror 7 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000641 1713208635.670628 read 0x004c61 thru 0x0077f7 (0x2b97 bytes) 000642 1713208635.672358 mapwrite 0x02ae07 thru 0x035272 (0xa46c bytes) 000646 1713208635.702917 mapread 0x024699 thru 0x02917c (0x4ae4 bytes) 000647 1713208635.705466 write 0x000aa9 thru 0x001f1f (0x1477 bytes) 000648 1713208635.707624 trunc from 0x035273 to 0x010987 (0x248eb bytes) 000649 1713208635.711512 trunc from 0x010988 to 0x014b30 (0x41a9 bytes) 000651 1713208635.715673 trunc from 0x014b31 to 0x00451f (0x10611 bytes) 000653 1713208635.720213 trunc from 0x004520 to 0x00852a (0x400b bytes) 000654 1713208635.724314 trunc from 0x00852b to 0x03189a (0x29370 bytes) 000655 1713208635.727752 mapread 0x0171ad thru 0x01c313 (0x5167 bytes) 000656 1713208635.731942 MIRROR_EXTEND to 15 mirrors 000657 1713208635.845289 read 0x014959 thru 0x0191c7 (0x486f bytes) 000659 1713208635.849909 MIRROR_SPLIT mirror 3 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000660 1713208635.907086 mapwrite 0x01390c thru 0x01f25b (0xb950 bytes) 000661 1713208635.962536 mapwrite 0x01865b thru 0x01bf92 (0x3938 bytes) 000662 1713208635.977450 trunc from 0x03189b to 0x018696 (0x19204 bytes) 000665 1713208635.985939 write 0x014b2a thru 0x01d98a (0x8e61 bytes) 000666 1713208635.986678 read 0x0170f1 thru 0x017634 (0x544 bytes) 000667 1713208635.988047 MIRROR_SPLIT mirror 2 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000669 1713208636.033572 read 0x018687 thru 0x01d98a (0x5304 bytes) 000670 1713208636.036022 trunc from 0x01d98b to 0x01ebce (0x1244 bytes) 000671 1713208636.039525 write 0x006653 thru 0x008994 (0x2342 bytes) 000673 1713208636.041848 mapread 0x018f35 thru 0x01ebce (0x5c9a bytes) 000675 1713208636.044113 mapread 0x00bfc9 thru 0x015a61 (0x9a99 bytes) 000676 1713208636.047650 write 0x039b0d thru 0x03ffff (0x64f3 bytes) 000677 1713208636.048337 read 0x02aa29 thru 0x02ebe6 (0x41be bytes) 000678 1713208636.050540 mapread 0x03841b thru 0x03ffff (0x7be5 bytes) 000679 1713208636.050977 mapwrite 0x03d288 thru 0x03ffff (0x2d78 bytes) 000680 1713208636.065211 write 0x0166fa thru 0x023e1e (0xd725 bytes) 000681 1713208636.065942 mapread 0x0345a8 thru 0x037416 (0x2e6f bytes) 000682 1713208636.066351 trunc from 0x040000 to 0x03779c (0x8863 bytes) 000683 1713208636.070269 trunc from 0x03779d to 0x01bef0 (0x1b8ac bytes) 000684 1713208636.074602 write 0x009a42 thru 0x011db0 (0x836f bytes) 000686 1713208636.075550 write 0x038e3c thru 0x03dabf (0x4c84 bytes) 000687 1713208636.076117 mapwrite 0x03f91e thru 0x03ffff (0x6e2 bytes) 000688 1713208636.094355 read 0x015549 thru 0x019468 (0x3f20 bytes) 000689 1713208636.094671 mapwrite 0x026f8b thru 0x03689a (0xf910 bytes) 000690 1713208636.110453 mapwrite 0x02926a thru 0x033f9b (0xad32 bytes) 000692 1713208636.123493 mapread 0x03c855 thru 0x03ffff (0x37ab bytes) 000694 1713208636.124099 mapwrite 0x039421 thru 0x03ffff (0x6bdf bytes) 000695 1713208636.136874 mapread 0x008ec3 thru 0x0115dd (0x871b bytes) 000696 1713208636.137546 mapwrite 0x022577 thru 0x029d8d (0x7817 bytes) 000697 1713208636.150791 read 0x019fe6 thru 0x028625 (0xe640 bytes) 000699 1713208636.152625 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000700 1713208636.195486 mapwrite 0x01bc38 thru 0x020500 (0x48c9 bytes) 000702 1713208636.216104 trunc from 0x040000 to 0x00a2e1 (0x35d1e bytes) 000703 1713208636.221121 MIRROR_EXTEND to 13 mirrors 000704 1713208636.319543 trunc from 0x00a2e2 to 0x00ac2b (0x94a bytes) 000706 1713208636.340533 read 0x0087ec thru 0x00ac2b (0x2440 bytes) 000707 1713208636.342344 read 0x00161e thru 0x00ac2b (0x960e bytes) 000708 1713208636.344462 mapread 0x004401 thru 0x0090b1 (0x4cb1 bytes) 000709 1713208636.344946 read 0x008ee0 thru 0x00ac2b (0x1d4c bytes) 000710 1713208636.345213 write 0x03e02f thru 0x03ffff (0x1fd1 bytes) 000713 1713208636.346356 write 0x00b226 thru 0x019c11 (0xe9ec bytes) 000715 1713208636.349572 trunc from 0x040000 to 0x038196 (0x7e69 bytes) 000716 1713208636.354834 MIRROR_EXTEND to 14 mirrors 000717 1713208636.467919 MIRROR_EXTEND to 15 mirrors 000718 1713208636.575653 mapread 0x00a2f6 thru 0x00aa5e (0x769 bytes) 000719 1713208636.579800 MIRROR_EXTEND to 16 mirrors 000721 1713208636.702047 mapwrite 0x0152ad thru 0x01d28a (0x7fde bytes) 000727 1713208636.753703 read 0x031b85 thru 0x038196 (0x6612 bytes) 000728 1713208636.756007 write 0x02089a thru 0x02170a (0xe71 bytes) 000730 1713208636.758450 mapread 0x016730 thru 0x0198c1 (0x3192 bytes) 000731 1713208636.758884 trunc from 0x038197 to 0x03caf2 (0x495c bytes) 000735 1713208636.769689 MIRROR_SPLIT mirror 7 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000736 1713208636.818605 mapread 0x0095e1 thru 0x00f1c9 (0x5be9 bytes) 000738 1713208636.824474 MIRROR_EXTEND to 16 mirrors 000739 1713208636.936002 read 0x0226a2 thru 0x02defd (0xb85c bytes) 000740 1713208636.939409 mapread 0x02d97e thru 0x0312bb (0x393e bytes) 000741 1713208636.945317 read 0x00d998 thru 0x01904a (0xb6b3 bytes) 000742 1713208636.948453 write 0x0112f6 thru 0x01c45f (0xb16a bytes) 000744 1713208636.966002 mapread 0x017ad7 thru 0x01c541 (0x4a6b bytes) 000747 1713208636.967166 mapread 0x0248d6 thru 0x0320c1 (0xd7ec bytes) 000749 1713208636.983394 mapwrite 0x0179b4 thru 0x01daf0 (0x613d bytes) 000751 1713208637.000455 write 0x00125a thru 0x009d5e (0x8b05 bytes) 000753 1713208637.003423 mapwrite 0x01c4b5 thru 0x029896 (0xd3e2 bytes) 000754 1713208637.023982 mapread 0x001057 thru 0x008f02 (0x7eac bytes) 000755 1713208637.025936 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000757 1713208637.078494 write 0x0207da thru 0x021979 (0x11a0 bytes) 000758 1713208637.081319 read 0x005941 thru 0x00c0f2 (0x67b2 bytes) 000759 1713208637.083487 mapread 0x01a953 thru 0x025ffb (0xb6a9 bytes) 000761 1713208637.089530 MIRROR_RESYNC 15 mirrors 000765 1713208637.479698 write 0x02afa5 thru 0x039337 (0xe393 bytes) 000768 1713208637.497989 MIRROR_RESYNC 15 mirrors 000769 1713208637.877933 read 0x033b01 thru 0x03c1bd (0x86bd bytes) 000771 1713208637.880177 mapwrite 0x0043df thru 0x0056df (0x1301 bytes) 000774 1713208637.927915 MIRROR_EXTEND to 16 mirrors 000775 1713208638.034240 write 0x03f36f thru 0x03ffff (0xc91 bytes) 000777 1713208638.050036 trunc from 0x040000 to 0x02752d (0x18ad2 bytes) 000779 1713208638.058831 read 0x0249e6 thru 0x0255ba (0xbd5 bytes) 000780 1713208638.060631 mapread 0x01780f thru 0x0245a8 (0xcd9a bytes) 000781 1713208638.067099 write 0x00d676 thru 0x014fba (0x7945 bytes) 000782 1713208638.070291 write 0x006c3b thru 0x014e56 (0xe21c bytes) 000787 1713208638.072163 read 0x012d73 thru 0x015c98 (0x2f26 bytes) 000789 1713208638.072799 read 0x02453c thru 0x02752d (0x2ff2 bytes) 000790 1713208638.073120 read 0x009143 thru 0x009346 (0x204 bytes) 000792 1713208638.075467 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000793 1713208638.133373 trunc from 0x02752e to 0x024c2f (0x28fe bytes) 000795 1713208638.138282 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000796 1713208638.185557 mapread 0x001117 thru 0x00aa83 (0x996d bytes) 000797 1713208638.190661 mapread 0x003db8 thru 0x00bef2 (0x813b bytes) 000801 1713208638.191992 trunc from 0x024c30 to 0x00af74 (0x19cbb bytes) 000803 1713208638.196889 mapwrite 0x02c8ed thru 0x03721d (0xa931 bytes) 000805 1713208638.221690 write 0x01390f thru 0x0152f6 (0x19e8 bytes) 000807 1713208638.224511 mapread 0x017bd3 thru 0x01dc55 (0x6083 bytes) 000808 1713208638.226829 trunc from 0x03721e to 0x00c939 (0x2a8e4 bytes) 000810 1713208638.230929 write 0x03f591 thru 0x03ffff (0xa6f bytes) 000812 1713208638.231810 write 0x039a53 thru 0x03efd7 (0x5585 bytes) 000813 1713208638.234283 mapwrite 0x016345 thru 0x0174d2 (0x118e bytes) 000815 1713208638.251004 mapread 0x02e045 thru 0x03d4a3 (0xf45f bytes) 000816 1713208638.253500 write 0x01b330 thru 0x026987 (0xb658 bytes) 000817 1713208638.255323 mapwrite 0x010a5b thru 0x01706e (0x6614 bytes) 000818 1713208638.268261 trunc from 0x040000 to 0x03472b (0xb8d4 bytes) 000819 1713208638.271882 write 0x004537 thru 0x004edc (0x9a6 bytes) 000820 1713208638.272431 mapwrite 0x0083a1 thru 0x00ddd5 (0x5a35 bytes) 000822 1713208638.289651 read 0x020436 thru 0x022666 (0x2231 bytes) 000825 1713208638.290892 mapwrite 0x001424 thru 0x00ee2f (0xda0c bytes) 000827 1713208638.307328 read 0x026a00 thru 0x03472b (0xdd2c bytes) 000828 1713208638.311148 MIRROR_EXTEND to 15 mirrors 000829 1713208638.428550 mapread 0x02cc32 thru 0x03472b (0x7afa bytes) 000831 1713208638.435618 mapwrite 0x011cb2 thru 0x0193b2 (0x7701 bytes) 000833 1713208638.485875 mapwrite 0x005b96 thru 0x00a24f (0x46ba bytes) 000837 1713208638.502771 read 0x002c21 thru 0x006f7b (0x435b bytes) 000838 1713208638.503272 write 0x03b818 thru 0x03ebd8 (0x33c1 bytes) 000839 1713208638.503867 read 0x00ed13 thru 0x013438 (0x4726 bytes) 000840 1713208638.504145 mapread 0x0398f5 thru 0x03ebd8 (0x52e4 bytes) 000841 1713208638.506976 mapread 0x0150ee thru 0x02116d (0xc080 bytes) 000844 1713208638.507985 mapread 0x016e6c thru 0x022560 (0xb6f5 bytes) 000845 1713208638.508429 read 0x03b3c1 thru 0x03ebd8 (0x3818 bytes) 000847 1713208638.508902 mapread 0x01b0e4 thru 0x02297a (0x7897 bytes) 000848 1713208638.510844 MIRROR_RESYNC 15 mirrors 000849 1713208638.905804 mapread 0x035d0d thru 0x03ebd8 (0x8ecc bytes) 000850 1713208638.910395 write 0x00dafc thru 0x01867b (0xab80 bytes) 000852 1713208638.929329 mapread 0x021511 thru 0x02e659 (0xd149 bytes) truncating to largest ever: 0x3fdc1 000853 1713208638.933516 trunc from 0x03ebd9 to 0x03fdc0 (0x11e8 bytes) 000856 1713208638.943144 mapread 0x01857c thru 0x01889e (0x323 bytes) 000857 1713208638.943754 read 0x0326b1 thru 0x03e3fa (0xbd4a bytes) 000859 1713208638.946508 read 0x003447 thru 0x011419 (0xdfd3 bytes) 000860 1713208638.950175 MIRROR_SPLIT mirror 14 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000861 1713208639.001711 MIRROR_SPLIT mirror 11 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000862 1713208639.048614 mapread 0x009d7b thru 0x019aec (0xfd72 bytes) 000863 1713208639.055268 mapwrite 0x03716e thru 0x03ffff (0x8e92 bytes) 000864 1713208639.079822 MIRROR_EXTEND to 14 mirrors 000867 1713208639.210104 write 0x02f42b thru 0x03b889 (0xc45f bytes) 000869 1713208639.229550 mapwrite 0x036c9a thru 0x0383f3 (0x175a bytes) 000871 1713208639.245410 mapwrite 0x03e5bf thru 0x03ffff (0x1a41 bytes) 000876 1713208639.259861 mapwrite 0x00a57f thru 0x00b102 (0xb84 bytes) 000877 1713208639.273167 mapwrite 0x03ea41 thru 0x03ffff (0x15bf bytes) 000879 1713208639.285300 mapread 0x01aaa7 thru 0x028df5 (0xe34f bytes) 000880 1713208639.289632 mapread 0x011ea0 thru 0x01e9ec (0xcb4d bytes) 000882 1713208639.292430 mapread 0x0338a9 thru 0x03d994 (0xa0ec bytes) 000883 1713208639.293130 write 0x030ab0 thru 0x03cf74 (0xc4c5 bytes) 000884 1713208639.294083 read 0x03f3aa thru 0x03ffff (0xc56 bytes) 000885 1713208639.295926 MIRROR_RESYNC 14 mirrors 000888 1713208639.600705 mapread 0x023bd5 thru 0x03062c (0xca58 bytes) 000889 1713208639.607593 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000890 1713208639.699482 mapread 0x008880 thru 0x0185ed (0xfd6e bytes) 000891 1713208639.706437 MIRROR_RESYNC 13 mirrors 000893 1713208639.776331 read 0x01cdf5 thru 0x02301e (0x622a bytes) 000894 1713208639.778327 mapwrite 0x000ddc thru 0x008d04 (0x7f29 bytes) 000896 1713208639.825980 trunc from 0x040000 to 0x0295f2 (0x16a0d bytes) 000898 1713208639.834101 trunc from 0x0295f3 to 0x03a8ee (0x112fc bytes) 000899 1713208639.837347 mapwrite 0x01b698 thru 0x020f3a (0x58a3 bytes) 000900 1713208639.855007 MIRROR_RESYNC 13 mirrors 000901 1713208640.193163 mapread 0x035698 thru 0x038da7 (0x3710 bytes) 000902 1713208640.197288 mapread 0x004182 thru 0x010c0a (0xca89 bytes) 000903 1713208640.201275 read 0x004913 thru 0x005a3e (0x112c bytes) 000904 1713208640.202427 MIRROR_RESYNC 13 mirrors 000906 1713208640.268805 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000907 1713208640.355887 mapwrite 0x03d647 thru 0x03ffff (0x29b9 bytes) 000908 1713208640.390695 read 0x02cf6d thru 0x039296 (0xc32a bytes) 000910 1713208640.393353 trunc from 0x040000 to 0x01de74 (0x2218b bytes) 000912 1713208640.398129 read 0x010e34 thru 0x01cd3d (0xbf0a bytes) 000913 1713208640.400477 mapread 0x014179 thru 0x01ca10 (0x8898 bytes) 000914 1713208640.401078 mapwrite 0x01d98a thru 0x025570 (0x7be7 bytes) 000915 1713208640.419906 write 0x0295db thru 0x02a5a7 (0xfcd bytes) 000916 1713208640.420554 mapwrite 0x03e0bb thru 0x03ffff (0x1f45 bytes) 000917 1713208640.440657 mapread 0x0020f6 thru 0x00932d (0x7238 bytes) 000919 1713208640.441456 write 0x015764 thru 0x022823 (0xd0c0 bytes) 000920 1713208640.442407 read 0x007a53 thru 0x009562 (0x1b10 bytes) 000921 1713208640.442705 mapread 0x006804 thru 0x0125b5 (0xbdb2 bytes) 000925 1713208640.443713 trunc from 0x040000 to 0x03f045 (0xfba bytes) 000928 1713208640.448334 trunc from 0x03f046 to 0x035494 (0x9bb1 bytes) 000931 1713208640.452453 read 0x02152c thru 0x02fccd (0xe7a2 bytes) 000932 1713208640.452985 mapwrite 0x003879 thru 0x011a09 (0xe191 bytes) 000933 1713208640.469019 mapread 0x013c92 thru 0x021b4f (0xdebe bytes) 000943 1713208640.471371 mapread 0x00d524 thru 0x0173f7 (0x9ed4 bytes) 000948 1713208640.472653 mapread 0x010cd1 thru 0x011532 (0x862 bytes) 000949 1713208640.473007 trunc from 0x035495 to 0x021896 (0x13bfe bytes) 000950 1713208640.477216 mapread 0x016216 thru 0x01c6ef (0x64da bytes) 000951 1713208640.477879 write 0x032ff2 thru 0x0356d0 (0x26df bytes) 000952 1713208640.479860 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000953 1713208640.528876 write 0x0011de thru 0x00aa2d (0x9850 bytes) 000957 1713208640.534355 read 0x031693 thru 0x0356d0 (0x403e bytes) 000958 1713208640.537177 trunc from 0x0356d1 to 0x020839 (0x14e97 bytes) 000960 1713208640.541894 read 0x0030ef thru 0x00ab4d (0x7a5f bytes) 000962 1713208640.542562 mapwrite 0x009ec9 thru 0x011304 (0x743c bytes) 000964 1713208640.560499 mapread 0x01dd81 thru 0x020839 (0x2ab9 bytes) 000966 1713208640.561547 trunc from 0x02083a to 0x007b70 (0x18cc9 bytes) 000968 1713208640.567123 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000969 1713208640.612695 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) 000971 1713208640.655032 write 0x01b6e7 thru 0x01b75e (0x78 bytes) 000972 1713208640.655694 mapread 0x016771 thru 0x01b346 (0x4bd6 bytes) 000973 1713208640.661707 MIRROR_RESYNC 9 mirrors 000974 1713208640.895471 mapread 0x0181bd thru 0x01b75e (0x35a2 bytes) 000975 1713208640.899575 read 0x00986f thru 0x00fd5a (0x64ec bytes) 000976 1713208640.901539 mapread 0x0038a2 thru 0x00a12d (0x688c bytes) 000978 1713208640.903992 read 0x005438 thru 0x010de1 (0xb9aa bytes) 000979 1713208640.904226 write 0x0314b0 thru 0x039a13 (0x8564 bytes) 000980 1713208640.918469 read 0x030809 thru 0x0387bc (0x7fb4 bytes) 000981 1713208640.922402 MIRROR_EXTEND to 10 mirrors 000983 1713208641.054599 mapread 0x0039be thru 0x006f67 (0x35aa bytes) 000986 1713208641.063473 write 0x00abdf thru 0x011273 (0x6695 bytes) 000988 1713208641.082331 MIRROR_EXTEND to 11 mirrors 000989 1713208641.199387 mapread 0x015748 thru 0x025609 (0xfec2 bytes) 000991 1713208641.210662 mapread 0x02439d thru 0x026b3b (0x279f bytes) 000993 1713208641.215853 mapread 0x0293ba thru 0x037f7f (0xebc6 bytes) 000994 1713208641.222501 mapread 0x02d78a thru 0x036e44 (0x96bb bytes) 000995 1713208641.226292 mapread 0x0220c5 thru 0x02f001 (0xcf3d bytes) 000996 1713208641.231226 MIRROR_EXTEND to 12 mirrors 000997 1713208641.349487 mapread 0x02ab2d thru 0x02fe51 (0x5325 bytes) 000999 1713208641.353928 mapwrite 0x018ca9 thru 0x019f44 (0x129c bytes) 001000 1713208641.403906 mapwrite 0x00825a thru 0x00b0fe (0x2ea5 bytes) All operations completed A-OK! PASS 70a (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200a: stress test ===================== 15:17:23 (1713208643) Starting client: oleg239-client.virtnet: -o user_xattr,flock oleg239-server@tcp:/lustre /mnt/lustre2 Starting client: oleg239-client.virtnet: -o user_xattr,flock oleg239-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5994176 .. Extending file size to 369760 .. failed Extending file size to 4527136 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6812480 .. failed Extending file size to 7164096 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8148256 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8805664 .. Extending file size to 7797216 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8617600 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8995392 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8997888 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9209344 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8842400 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9035264 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9078368 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 7075 7076 7078 7080 7082 192.168.202.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg239-client.virtnet /mnt/lustre2 (opts:) 192.168.202.139@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg239-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:18:29 (1713208709) Starting client: oleg239-client.virtnet: -o user_xattr,flock oleg239-server@tcp:/lustre /mnt/lustre2 Starting client: oleg239-client.virtnet: -o user_xattr,flock oleg239-server@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2630848 .. Extending file size to 2095648 .. done Extending file size to 6359968 .. Extending file size to 5374400 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6973312 .. Extending file size to 7297792 .. Extending file size to 8567744 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7555104 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8165056 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8721184 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8664832 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8813440 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8774528 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9072928 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9025472 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9227616 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 10465 10466 10467 10469 10472 failed 192.168.202.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg239-client.virtnet /mnt/lustre2 (opts:) 192.168.202.139@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg239-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:19:34 (1713208774) Starting client: oleg239-client.virtnet: -o user_xattr,flock oleg239-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5248 2203264 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5248 2203264 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0398997 s, 52.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0763835 s, 27.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg239-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:19:40 (1713208780) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0320262 s, 65.5 MB/s PASS 202 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 203: mirror file preserve mirror ID === 15:19:43 (1713208783) lfs mirror mirror: cannot get UNLOCK lease, ext 8: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd74:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd74:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd75:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xd77:0x0] } PASS 203 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:19:47 (1713208787) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0236869 s, 88.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd76:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd78:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd79:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:19:50 (1713208790) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0272538 s, 76.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd77:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd78:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd77:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd78:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7b:0x0] } PASS 204b (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:19:53 (1713208793) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0269045 s, 77.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd79:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd79:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7d:0x0] } PASS 204c (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:20:18 (1713208818) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg239-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146745 s, 71.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7b:0x0] } PASS 204d (40s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:20:59 (1713208859) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg239-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0285145 s, 73.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd80:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd80:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:21:45 (1713208905) sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg239-server mds-ost sync done. Creating new pool oleg239-server: Pool lustre.test_204f created Adding targets to pool oleg239-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg239-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0229249 s, 91.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd81:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd81:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg239-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg239-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg239-server: Pool lustre.test_204f destroyed PASS 204f (54s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 205: lfs mirror extend to set prefer and nocompr flags ========================================================== 15:22:41 (1713208961) PASS 205 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:22:44 (1713208964) oleg239-server: Pool lustre.test_206 created Adding targets to pool oleg239-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg239-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg239-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg239-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg239-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (14s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:22:59 (1713208979) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187576 s, 55.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.008561 s, 122 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x620280f9 Mirror 3: 0x620280f9 PASS 207 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:23:03 (1713208983) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:23:05 (1713208985) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:23:07 (1713208987) Starting client: oleg239-client.virtnet: -o user_xattr,flock oleg239-server@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0148792 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xd00000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1bd0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1bd0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1bd0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x850000 Read this (4 bytes): BBBB 192.168.202.139@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg239-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:23:22 (1713209002) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00254327 s, 3.2 MB/s the very first read read with warm cache PASS 209b (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210a: handle broken mirrored lovea ==== 15:23:25 (1713209005) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00844399 s, 124 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x72d lmm_fid: [0x200000404:0x72d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x72d lmm_fid: [0x200000404:0x72d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd87:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835757 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-04-15 15:23:25.000000000 -0400 Modify: 2024-04-15 15:23:25.000000000 -0400 Change: 2024-04-15 15:23:26.000000000 -0400 Birth: - PASS 210a (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:23:28 (1713209008) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-flr test complete, duration 770 sec ============ 15:23:29 (1713209009) === sanity-flr: start cleanup 15:23:30 (1713209010) === === sanity-flr: finish cleanup 15:23:30 (1713209010) ===