[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 511584132 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003124] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.007861] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010134] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014044] *** VALIDATE selinux *** [ 0.021578] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025428] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027090] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] *** VALIDATE tmpfs *** [ 0.029373] *** VALIDATE proc *** [ 0.030231] *** VALIDATE cgroup *** [ 0.031008] *** VALIDATE cgroup2 *** [ 0.033129] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036025] Spectre V2 : User space: Vulnerable [ 0.037004] Speculative Store Bypass: Vulnerable [ 0.039238] debug: unmapping init [mem 0xffffffffa0859000-0xffffffffa0860fff] [ 0.041257] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042718] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043017] ... version: 2 [ 0.044011] ... bit width: 48 [ 0.045009] ... generic registers: 4 [ 0.046010] ... value mask: 0000ffffffffffff [ 0.047013] ... max period: 00007fffffffffff [ 0.048012] ... fixed-purpose events: 3 [ 0.049012] ... event mask: 000000070000000f [ 0.050325] rcu: Hierarchical SRCU implementation. [ 0.052254] smp: Bringing up secondary CPUs ... [ 0.053602] x86: Booting SMP configuration: [ 0.054023] .... node #0, CPUs: #1 #2 #3 [ 0.061251] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064040] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.103058] node 0 deferred pages initialised in 37ms [ 0.106369] devtmpfs: initialized [ 0.107217] x86/mm: Memory block size: 128MB [ 0.110183] gcov: version magic: 0x41383552 [ 0.112338] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.113087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.114485] pinctrl core: initialized pinctrl subsystem [ 0.115442] [ 0.116000] ************************************************************* [ 0.116019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.117016] ** ** [ 0.118018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.119013] ** ** [ 0.120011] ** This means that this kernel is built to expose internal ** [ 0.121013] ** IOMMU data structures, which may compromise security on ** [ 0.122013] ** your system. ** [ 0.123014] ** ** [ 0.124011] ** If you see this message and you are not debugging the ** [ 0.125013] ** kernel, report this immediately to your vendor! ** [ 0.126013] ** ** [ 0.127014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.128066] ************************************************************* [ 0.129872] NET: Registered protocol family 16 [ 0.130604] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.131061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.132131] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.133686] cpuidle: using governor menu [ 0.136138] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.139576] PCI: Using configuration type 1 for base access [ 0.141129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.151356] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.153044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.157293] cryptd: max_cpu_qlen set to 1000 [ 0.160060] ACPI: Added _OSI(Module Device) [ 0.161024] ACPI: Added _OSI(Processor Device) [ 0.164018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.166026] ACPI: Added _OSI(Processor Aggregator Device) [ 0.171561] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.177482] ACPI: Interpreter enabled [ 0.179060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.180013] ACPI: Using IOAPIC for interrupt routing [ 0.182113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.186054] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.198083] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.201043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.203015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.206075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.211477] acpiphp: Slot [2] registered [ 0.213156] acpiphp: Slot [5] registered [ 0.214074] acpiphp: Slot [6] registered [ 0.215169] acpiphp: Slot [7] registered [ 0.217168] acpiphp: Slot [8] registered [ 0.218183] acpiphp: Slot [9] registered [ 0.220210] acpiphp: Slot [10] registered [ 0.221100] acpiphp: Slot [3] registered [ 0.223079] acpiphp: Slot [4] registered [ 0.224072] acpiphp: Slot [11] registered [ 0.225102] acpiphp: Slot [12] registered [ 0.227072] acpiphp: Slot [13] registered [ 0.228096] acpiphp: Slot [14] registered [ 0.229188] acpiphp: Slot [15] registered [ 0.231096] acpiphp: Slot [16] registered [ 0.232075] acpiphp: Slot [17] registered [ 0.234133] acpiphp: Slot [18] registered [ 0.235079] acpiphp: Slot [19] registered [ 0.236150] acpiphp: Slot [20] registered [ 0.239119] acpiphp: Slot [21] registered [ 0.240108] acpiphp: Slot [22] registered [ 0.241067] acpiphp: Slot [23] registered [ 0.242077] acpiphp: Slot [24] registered [ 0.243146] acpiphp: Slot [25] registered [ 0.245182] acpiphp: Slot [26] registered [ 0.246094] acpiphp: Slot [27] registered [ 0.247147] acpiphp: Slot [28] registered [ 0.249156] acpiphp: Slot [29] registered [ 0.250079] acpiphp: Slot [30] registered [ 0.252236] acpiphp: Slot [31] registered [ 0.253049] PCI host bridge to bus 0000:00 [ 0.254015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.256021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.258029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.261020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.263027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.266033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.268158] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.271156] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.277316] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.300035] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.306052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.310030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.314018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.318052] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.322859] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.327000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.331169] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.337612] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.343015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.366015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.377029] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.389841] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.398013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.404016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.430208] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.431000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.431000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.431000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.431000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.434000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.442014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.464029] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.499232] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.502000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.502000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.503000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.503000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.503000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.503000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.503000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.530073] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.539022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.547018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.566032] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.584896] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.593243] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.597544] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.600429] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.605484] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.692617] iommu: Default domain type: Passthrough [ 0.700428] SCSI subsystem initialized [ 0.702200] ACPI: bus type USB registered [ 0.703191] usbcore: registered new interface driver usbfs [ 0.705330] usbcore: registered new interface driver hub [ 0.707093] usbcore: registered new device driver usb [ 0.710145] pps_core: LinuxPPS API ver. 1 registered [ 0.712009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.715085] PTP clock support registered [ 0.717070] EDAC MC: Ver: 3.0.0 [ 0.718971] PCI: Using ACPI for IRQ routing [ 0.721036] NetLabel: Initializing [ 0.722011] NetLabel: domain hash size = 128 [ 0.724010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.727081] NetLabel: unlabeled traffic allowed by default [ 0.730326] vgaarb: loaded [ 0.732430] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.735015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.743470] clocksource: Switched to clocksource kvm-clock [ 0.864387] VFS: Disk quotas dquot_6.6.0 [ 0.865658] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.868267] *** VALIDATE ramfs *** [ 0.869558] *** VALIDATE hugetlbfs *** [ 0.871077] pnp: PnP ACPI init [ 0.873557] pnp: PnP ACPI: found 6 devices [ 0.891516] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.896086] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.899080] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.901769] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.904479] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.907410] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.909913] NET: Registered protocol family 2 [ 0.912519] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.918297] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.921976] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.927739] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.931373] TCP: Hash tables configured (established 65536 bind 65536) [ 0.934648] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.937937] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.941099] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.944738] NET: Registered protocol family 1 [ 0.947641] RPC: Registered named UNIX socket transport module. [ 0.950089] RPC: Registered udp transport module. [ 0.951995] RPC: Registered tcp transport module. [ 0.954099] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.956900] NET: Registered protocol family 44 [ 0.958864] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.961429] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.963856] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.966433] PCI: CLS 0 bytes, default 64 [ 0.969343] Unpacking initramfs... [ 2.627960] debug: unmapping init [mem 0xffff9ab2bcc54000-0xffff9ab2bffbffff] [ 2.631794] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.633669] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.636178] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.379631] Initialise system trusted keyrings [ 3.381617] Key type blacklist registered [ 3.386649] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.422901] zbud: loaded [ 3.426256] *** VALIDATE nfs *** [ 3.427594] *** VALIDATE nfs4 *** [ 3.429507] pstore: using deflate compression [ 3.435672] Platform Keyring initialized [ 3.555307] NET: Registered protocol family 38 [ 3.557377] Key type asymmetric registered [ 3.559164] Asymmetric key parser 'x509' registered [ 3.561319] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.564738] io scheduler mq-deadline registered [ 3.566655] io scheduler kyber registered [ 3.568476] io scheduler bfq registered [ 3.570435] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.573827] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.577349] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.580746] ACPI: Power Button [PWRF] [ 3.586758] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.596175] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.633630] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.640735] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.657737] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.686331] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.717475] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.722247] Non-volatile memory driver v1.3 [ 3.723811] Linux agpgart interface v0.103 [ 3.755704] virtio_blk virtio1: [vda] 145840 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.758505] vda: detected capacity change from 0 to 74670080 [ 3.780669] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.783804] vdb: detected capacity change from 0 to 1073741824 [ 3.834491] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.842709] vdc: detected capacity change from 0 to 2621440000 [ 3.876573] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.879480] vdd: detected capacity change from 0 to 2621440000 [ 3.935456] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.009759] vde: detected capacity change from 0 to 4294967296 [ 4.054816] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.067286] vdf: detected capacity change from 0 to 4294967296 [ 4.079308] libphy: Fixed MDIO Bus: probed [ 4.093289] usbcore: registered new interface driver usbserial_generic [ 4.097482] usbserial: USB Serial support registered for generic [ 4.101284] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.109314] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.112882] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.116471] mousedev: PS/2 mouse device common for all mice [ 4.122145] rtc_cmos 00:05: RTC can wake from S4 [ 4.127892] rtc_cmos 00:05: registered as rtc0 [ 4.133689] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.141332] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.141958] intel_pstate: CPU model not supported [ 4.151513] hid: raw HID events driver (C) Jiri Kosina [ 4.154000] usbcore: registered new interface driver usbhid [ 4.155818] usbhid: USB HID core driver [ 4.157743] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.158374] drop_monitor: Initializing network drop monitor service [ 4.166953] Initializing XFRM netlink socket [ 4.169245] NET: Registered protocol family 10 [ 4.172410] Segment Routing with IPv6 [ 4.174067] NET: Registered protocol family 17 [ 4.176567] mpls_gso: MPLS GSO support [ 4.178708] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.187285] RAS: Correctable Errors collector initialized. [ 4.189583] AVX version of gcm_enc/dec engaged. [ 4.191334] AES CTR mode by8 optimization enabled [ 4.326387] sched_clock: Marking stable (4326363137, 0)->(5719234937, -1392871800) [ 4.353750] registered taskstats version 1 [ 4.358691] Loading compiled-in X.509 certificates [ 4.366507] zswap: loaded using pool lzo/zbud [ 4.449286] Key type big_key registered [ 4.478350] Key type encrypted registered [ 4.480742] ima: No TPM chip found, activating TPM-bypass! [ 4.483806] ima: Allocated hash algorithm: sha1 [ 4.486410] ima: No architecture policies found [ 4.500385] evm: Initialising EVM extended attributes: [ 4.503312] evm: security.selinux [ 4.504866] evm: security.ima [ 4.506405] evm: security.capability [ 4.508158] evm: HMAC attrs: 0x1 [ 4.512379] rtc_cmos 00:05: setting system clock to 2026-08-10 13:48:58 UTC (1786369738) [ 4.523975] debug: unmapping init [mem 0xffffffffa1803000-0xffffffffa19fffff] [ 4.530045] debug: unmapping init [mem 0xffffffffa0582000-0xffffffffa0858fff] [ 4.541220] Write protecting the kernel read-only data: 28672k [ 4.548226] debug: unmapping init [mem 0xffffffff9ec03000-0xffffffff9edfffff] [ 4.554103] debug: unmapping init [mem 0xffffffff9f514000-0xffffffff9f5fffff] [ 4.666818] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.690862] systemd[1]: Detected virtualization kvm. [ 4.696262] systemd[1]: Detected architecture x86-64. [ 4.700936] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.753797] systemd[1]: No hostname configured. [ 4.762160] systemd[1]: Set hostname to . [ 4.768928] random: systemd: uninitialized urandom read (16 bytes read) [ 4.775431] systemd[1]: Initializing machine ID from random generator. [ 4.824365] random: ln: uninitialized urandom read (6 bytes read) [ 5.320751] random: systemd: uninitialized urandom read (16 bytes read) [ 5.324369] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.329260] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.339718] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.959984] device-mapper: uevent: version 1.0.3 [ 6.962989] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.614876] virtio_net virtio0 ens2: renamed from eth0 [ 9.668711] random: fast init done [ 10.081227] scsi host0: ata_piix [ 10.168395] scsi host1: ata_piix [ 10.169760] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.179926] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.053072] random: crng init done [ 15.058454] random: 7 urandom warning(s) missed due to ratelimiting [ 18.312377] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.461738] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ 19.800611] hrtimer: interrupt took 14087237 ns [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.517946] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.130585] SELinux: Disabled at runtime. [ 22.225540] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.237254] systemd[1]: Detected virtualization kvm. [ 22.239927] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.457578] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.463850] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.471986] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.480715] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.486284] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.494921] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.502973] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ 24.250451] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 25.289182] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.976451] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.087563] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.910759] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.065510] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 32.285482] Key type dns_resolver registered [ 32.710842] NFS: Registering the id_resolver key type [ 32.713261] Key type id_resolver registered [ 32.714993] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg348-server login: [ 84.972403] libcfs: loading out-of-tree module taints kernel. [ 85.015464] Key type ._llcrypt registered [ 85.019094] Key type .llcrypt registered [ 85.090707] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 101.818351] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 104.431996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 104.446786] alg: No test for adler32 (adler32-zlib) [ 105.702446] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 106.261807] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 107.961857] Key type lgssc registered [ 109.689937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 124.412407] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 156.320614] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 166.503263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 166.527513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.776446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 167.815263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 167.888427] Lustre: lustre-MDT0000: new disk, initializing [ 167.955297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.969451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 171.572073] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.624916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.698835] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 181.721685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 181.727585] Lustre: Skipped 1 previous similar message [ 181.784530] Lustre: lustre-MDT0001: new disk, initializing [ 181.833148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 181.860534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 181.877134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 185.060882] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.981624] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 195.442534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 195.607415] Lustre: lustre-OST0000: new disk, initializing [ 195.611180] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 195.616360] Lustre: 8447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 195.664804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.037579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 196.050157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 196.118156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.669942] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.710716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.803737] Lustre: lustre-OST0001: new disk, initializing [ 209.807516] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.812949] Lustre: 9520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.866383] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.940329] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.624795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 216.636119] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 216.667184] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 224.810574] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 229.196448] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 233.754644] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.627683] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing yml_node [ 241.264459] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 243.107163] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 245.055464] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 246.552727] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 10 09:52:59 EDT 2026 [ 258.794298] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 259.906571] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 273.888873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 273.896657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.902264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 277.985836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 277.988053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 277.995913] Lustre: Skipped 1 previous similar message [ 278.002855] Lustre: Skipped 1 previous similar message [ 279.632764] Lustre: server umount lustre-MDT0000 complete [ 281.952589] LustreError: 12158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370015 with bad export cookie 15852931718965514016 [ 281.954790] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 281.958564] LustreError: 12158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 283.106126] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.106345] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 283.118642] Lustre: Skipped 2 previous similar messages [ 283.125194] Lustre: Skipped 3 previous similar messages [ 286.114538] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 286.118903] Lustre: Skipped 1 previous similar message [ 288.235337] Lustre: server umount lustre-MDT0001 complete [ 290.798593] Lustre: server umount lustre-OST0000 complete [ 293.112752] Lustre: server umount lustre-OST0001 complete [ 297.978887] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 303.919812] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 325.693277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 325.835724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 325.850988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 325.902523] Lustre: lustre-MDT0000: new disk, initializing [ 325.949216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 325.960329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 329.449320] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 337.049348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 337.151299] Lustre: 15064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 337.198827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 337.201780] Lustre: Skipped 1 previous similar message [ 337.260883] Lustre: lustre-MDT0001: new disk, initializing [ 337.323704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 337.356200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 337.363996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 340.366542] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 346.893342] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 350.703629] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 354.963711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 355.158587] Lustre: lustre-OST0000: new disk, initializing [ 355.166612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 355.171270] Lustre: 17034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 355.246782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 356.814857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 356.822222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 356.919345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 359.524664] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 366.184946] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 369.050620] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 369.227777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 372.244157] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 372.407537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 377.318606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.319758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 377.324372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 377.324761] Lustre: Skipped 1 previous similar message [ 379.684764] Lustre: server umount lustre-OST0000 complete [ 382.434271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.449215] Lustre: Skipped 1 previous similar message [ 387.554506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 387.559739] Lustre: Skipped 4 previous similar messages [ 388.219389] Lustre: server umount lustre-MDT0000 complete [ 390.722375] LustreError: 17033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370124 with bad export cookie 15852931718965515549 [ 390.727660] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 390.738807] LustreError: 17033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 390.964788] Lustre: server umount lustre-MDT0001 complete [ 393.424417] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:55:26 (1786370126) [ 394.764498] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 395.793535] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:55:29 (1786370129) [ 399.632476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.909346] LustreError: 19447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 399.919910] LustreError: 19447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 399.954734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.353185] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 404.962469] LustreError: 19448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 407.395167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.110672] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 414.421685] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 417.752070] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 421.332715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 425.333147] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 429.718241] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 438.243337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 438.244813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.246977] LustreError: Skipped 1 previous similar message [ 438.247717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 438.252356] Lustre: Skipped 1 previous similar message [ 439.787571] Lustre: server umount lustre-OST0000 complete [ 443.362371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 448.121829] Lustre: server umount lustre-MDT0000 complete [ 448.488735] LustreError: 19447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 448.505967] LustreError: 19447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 450.288928] LustreError: 19429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370184 with bad export cookie 15852931718965516431 [ 450.290693] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 450.298692] LustreError: 19429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 450.502327] Lustre: server umount lustre-MDT0001 complete [ 456.829337] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 458.589763] Key type lgssc unregistered [ 458.793736] LNet: 22975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.799808] LNetError: 22975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 458.811755] LNet: Removed LNI 192.168.203.148@tcp [ 459.428216] Key type .llcrypt unregistered [ 459.430873] Key type ._llcrypt unregistered [ 473.203563] Key type ._llcrypt registered [ 473.205714] Key type .llcrypt registered [ 473.272908] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 473.741579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 473.816384] alg: No test for adler32 (adler32-zlib) [ 474.829255] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 475.013492] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 476.664548] Key type lgssc registered [ 477.459144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 482.261656] loop: module loaded [ 482.317124] loop0: detected capacity change from 0 to 204800000 [ 482.430026] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 482.436785] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 482.633534] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 486.257964] loop0: detected capacity change from 0 to 204800000 [ 486.372744] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 486.382384] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 486.392182] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 489.576645] loop0: detected capacity change from 0 to 204800000 [ 492.892495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 492.914759] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.037614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 494.073342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 494.121796] Lustre: lustre-MDT0000: new disk, initializing [ 494.173631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.188172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 496.740597] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 502.424906] loop1: detected capacity change from 0 to 204800000 [ 505.721146] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 505.910440] Lustre: lustre-OST07c6: new disk, initializing [ 505.913386] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 505.917405] Lustre: Skipped 1 previous similar message [ 505.922426] Lustre: 26134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 505.974892] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 508.911159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 508.917491] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 508.950889] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 509.486930] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 513.529318] Lustre: 25121:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 513.534470] Lustre: 25121:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 513.538262] Lustre: 25121:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 513.542186] Lustre: 25121:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 513.546770] Lustre: 25121:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 513.550816] Lustre: 25121:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 526.588481] Lustre: 25122:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 526.595191] Lustre: 25122:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 526.600016] Lustre: 25122:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 526.603871] Lustre: 25122:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 526.608261] Lustre: 25122:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 526.611993] Lustre: 25122:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 530.547511] Lustre: 25123:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 530.553268] Lustre: 25123:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 530.557791] Lustre: 25123:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 530.561638] Lustre: 25123:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 530.564988] Lustre: 25123:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 530.568838] Lustre: 25123:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 530.573117] Lustre: 25123:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 530.576962] Lustre: 25123:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 530.581391] Lustre: 25123:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 530.585142] Lustre: 25123:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 530.588466] Lustre: 25123:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 530.592378] Lustre: 25123:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 533.706635] Lustre: 25121:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 533.712240] Lustre: 25121:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 533.718794] Lustre: 25121:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 533.724796] Lustre: 25121:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 533.729865] Lustre: 25121:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 533.733671] Lustre: 25121:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 533.737148] Lustre: 25121:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 533.747761] Lustre: 25121:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 533.751625] Lustre: 25121:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 533.760372] Lustre: 25121:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 533.766978] Lustre: 25121:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 533.770770] Lustre: 25121:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.876830] Lustre: 25122:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 537.884605] Lustre: 25122:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 537.890240] Lustre: 25122:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 537.896190] Lustre: 25122:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.901969] Lustre: 25122:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 537.906446] Lustre: 25122:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.911257] Lustre: 25122:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 537.917338] Lustre: 25122:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.922200] Lustre: 25122:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 537.926443] Lustre: 25122:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.931284] Lustre: 25122:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 537.936545] Lustre: 25122:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 548.208115] Lustre: 25123:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 548.214808] Lustre: 25123:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 548.218438] Lustre: 25123:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 548.222361] Lustre: 25123:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 548.226836] Lustre: 25123:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 548.231269] Lustre: 25123:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 548.235618] Lustre: 25123:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 548.240281] Lustre: 25123:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 548.244979] Lustre: 25123:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 548.249315] Lustre: 25123:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 548.253559] Lustre: 25123:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 548.258218] Lustre: 25123:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 595.938961] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.939217] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 595.949108] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 599.520392] Lustre: server umount lustre-OST07c6 complete [ 605.625766] Lustre: server umount lustre-MDT0000 complete [ 616.088260] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 618.225486] Key type lgssc unregistered [ 618.412073] LNet: 28693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.418295] LNetError: 28693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 618.431684] LNet: Removed LNI 192.168.203.148@tcp [ 619.020608] Key type .llcrypt unregistered [ 619.023506] Key type ._llcrypt unregistered [ 630.945335] Key type ._llcrypt registered [ 630.946920] Key type .llcrypt registered [ 631.023553] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 631.466183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 631.474902] alg: No test for adler32 (adler32-zlib) [ 632.400138] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 632.524832] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 634.144156] Key type lgssc registered [ 634.813993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 639.313744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 639.325396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.533286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 643.186528] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 648.427412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.656296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 651.087960] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 655.296376] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 658.166944] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 661.833490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 662.018492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.136676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 665.333402] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 669.434386] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 683.491036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 683.492050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 683.509511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 683.514258] Lustre: Skipped 1 previous similar message [ 688.614535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 688.619030] Lustre: Skipped 1 previous similar message [ 689.534900] Lustre: server umount lustre-OST0000 complete [ 693.729259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.731432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 693.738315] Lustre: Skipped 2 previous similar messages [ 697.997321] Lustre: server umount lustre-MDT0000 complete [ 698.849172] LustreError: 29957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 698.862523] LustreError: 29957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 700.071969] LustreError: 29937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370434 with bad export cookie 16344177434403465147 [ 700.076434] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 700.080568] LustreError: 29937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 700.259738] Lustre: server umount lustre-MDT0001 complete [ 705.830232] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 707.395428] Key type lgssc unregistered [ 707.558749] LNet: 33695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.563868] LNetError: 33695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.577524] LNet: Removed LNI 192.168.203.148@tcp [ 708.014823] Key type .llcrypt unregistered [ 708.016606] Key type ._llcrypt unregistered [ 718.418460] Key type ._llcrypt registered [ 718.420263] Key type .llcrypt registered [ 718.478454] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 719.074892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 719.101985] alg: No test for adler32 (adler32-zlib) [ 720.065216] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 720.225552] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 721.832160] Key type lgssc registered [ 722.518881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 726.946898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 726.959875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 728.219865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 730.619327] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 735.021463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.233817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 737.261496] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 741.125523] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 744.037201] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 747.839154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 748.042560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.461169] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 753.206267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 756.252348] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 768.481685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 768.485360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 768.491968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 770.530716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 770.531490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 770.537610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 772.465521] Lustre: server umount lustre-OST0000 complete [ 775.650641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.652073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 775.658564] Lustre: Skipped 1 previous similar message [ 775.665295] Lustre: Skipped 2 previous similar messages [ 780.720985] Lustre: server umount lustre-MDT0000 complete [ 780.771170] LustreError: 34950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 780.783223] LustreError: 34950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 782.747674] LustreError: 34924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370516 with bad export cookie 6413611468694114967 [ 782.748342] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 782.760836] LustreError: 34924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 782.947134] Lustre: server umount lustre-MDT0001 complete [ 789.183637] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 790.952659] Key type lgssc unregistered [ 791.122172] LNet: 38489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.128369] LNetError: 38489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.139763] LNet: Removed LNI 192.168.203.148@tcp [ 791.613098] Key type .llcrypt unregistered [ 791.615520] Key type ._llcrypt unregistered [ 804.774157] Key type ._llcrypt registered [ 804.775573] Key type .llcrypt registered [ 804.822879] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 805.217079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 805.245082] alg: No test for adler32 (adler32-zlib) [ 806.159933] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 806.274828] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 807.872198] Key type lgssc registered [ 808.516129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 812.581434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 812.592341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 813.833541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 816.364672] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 821.527742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 821.751846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 824.745283] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 829.256199] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 832.891492] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 836.831603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 837.063470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 838.139976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 840.384960] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 844.324133] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 851.429225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.431045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 851.438065] Lustre: Skipped 1 previous similar message [ 851.559845] Lustre: server umount lustre-MDT0000 complete [ 852.130205] LustreError: 42321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 852.140264] LustreError: 42321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 853.780049] LustreError: 39864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370587 with bad export cookie 8105239300620447136 [ 853.783386] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 853.789091] LustreError: 39864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 856.545698] LustreError: 39880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 856.547505] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.570198] Lustre: Skipped 1 previous similar message [ 856.573189] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 856.578868] Lustre: Skipped 2 previous similar messages [ 858.593894] LustreError: 41821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 858.593944] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 858.603410] LustreError: 41821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 858.608740] Lustre: Skipped 1 previous similar message [ 862.340329] LustreError: 42321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 862.340439] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.48@tcp (stopping) [ 867.459102] LustreError: 39880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 867.476127] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.48@tcp (stopping) [ 867.476135] Lustre: Skipped 1 previous similar message [ 867.483533] LustreError: 39880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 868.320616] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 868.435732] Lustre: server umount lustre-MDT0001 complete [ 881.326600] Lustre: server umount lustre-OST0000 complete [ 890.122798] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 891.730687] Key type lgssc unregistered [ 891.934591] LNet: 43741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.940568] LNetError: 43741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.956936] LNet: Removed LNI 192.168.203.148@tcp [ 892.461443] Key type .llcrypt unregistered [ 892.463520] Key type ._llcrypt unregistered [ 907.130703] Key type ._llcrypt registered [ 907.133698] Key type .llcrypt registered [ 907.216513] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 907.909927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 907.920411] alg: No test for adler32 (adler32-zlib) [ 908.870551] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 909.041434] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 910.672929] Key type lgssc registered [ 911.445634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 916.359093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 916.376849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.611445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.231809] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 925.338433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.738214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 928.621594] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 933.358390] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 936.882452] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 940.873161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 941.068609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 942.132912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 945.132679] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 949.709530] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 955.362594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 955.364476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.377167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 956.054330] Lustre: lustre-OST0000: Not available for connect from 192.168.203.48@tcp (stopping) [ 956.059205] Lustre: Skipped 1 previous similar message [ 960.292730] Lustre: server umount lustre-OST0000 complete [ 975.778586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.780670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 975.786524] Lustre: Skipped 2 previous similar messages [ 980.961534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 980.965734] Lustre: Skipped 2 previous similar messages [ 986.082804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 986.086270] Lustre: Skipped 1 previous similar message [ 989.152124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 989.231637] Lustre: server umount lustre-MDT0000 complete [ 991.201382] LustreError: 45136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 991.216290] LustreError: 45136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 991.342532] LustreError: 47713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370725 with bad export cookie 11347023975933073200 [ 991.345055] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 991.358835] LustreError: 47713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 997.616947] Lustre: server umount lustre-MDT0001 complete [ 1003.700410] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 1005.614579] Key type lgssc unregistered [ 1005.803967] LNet: 48847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.812207] LNetError: 48847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.824516] LNet: Removed LNI 192.168.203.148@tcp [ 1006.384218] Key type .llcrypt unregistered [ 1006.388866] Key type ._llcrypt unregistered [ 1020.589104] Key type ._llcrypt registered [ 1020.594378] Key type .llcrypt registered [ 1020.672965] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 1021.183847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1021.238227] alg: No test for adler32 (adler32-zlib) [ 1022.148737] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 1022.290588] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 1023.896171] Key type lgssc registered [ 1024.513895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1028.425294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1028.440895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.707547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1032.181413] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1036.935652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1037.143148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1039.482551] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1043.645294] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1047.124873] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1051.058333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1051.267347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1054.313709] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1054.384393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1058.073252] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1066.477505] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1068.404821] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1079.625560] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1010h06m52s [ 1080.525492] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1083.368395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1083.373544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.383263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.451610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.454930] Lustre: Skipped 4 previous similar messages [ 1086.098853] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.48@tcp (stopping) [ 1087.882418] Lustre: server umount lustre-MDT0000 complete [ 1089.696647] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1090.536740] LustreError: 50969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1090.556068] LustreError: 50969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1092.227819] LustreError: 51722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1093.410248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.495687] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1093.643916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.688598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1094.696526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1096.058368] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1097.864981] Lustre: DEBUG MARKER: Wait for df (36640) ... [ 1098.610610] Lustre: DEBUG MARKER: done [ 1103.840984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1103.841499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.847426] LustreError: Skipped 1 previous similar message [ 1103.848508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1103.856735] Lustre: Skipped 3 previous similar messages [ 1103.865942] Lustre: Skipped 2 previous similar messages [ 1106.335323] Lustre: server umount lustre-OST0000 complete [ 1109.985070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1109.986633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1109.990074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1109.994566] Lustre: Skipped 1 previous similar message [ 1109.997908] Lustre: Skipped 2 previous similar messages [ 1114.425919] Lustre: server umount lustre-MDT0000 complete [ 1115.043196] LustreError: 50243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.056316] LustreError: 50243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1116.169316] LustreError: 52866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370850 with bad export cookie 9597458009629259375 [ 1116.172772] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1116.175508] LustreError: 52866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1116.329850] Lustre: server umount lustre-MDT0001 complete [ 1120.973630] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 1122.227935] Key type lgssc unregistered [ 1122.386580] LNet: 55158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.391941] LNetError: 55158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1122.401591] LNet: Removed LNI 192.168.203.148@tcp [ 1122.847822] Key type .llcrypt unregistered [ 1122.849631] Key type ._llcrypt unregistered [ 1143.785861] Key type ._llcrypt registered [ 1143.787485] Key type .llcrypt registered [ 1143.844389] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 1144.367547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1144.385125] alg: No test for adler32 (adler32-zlib) [ 1145.318804] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 1145.462103] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 1147.080664] Key type lgssc registered [ 1147.622260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1151.733653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1151.743703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.888369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1152.901844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1152.989221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1155.226252] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1159.615082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.671266] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1159.689115] Lustre: 57341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1159.832141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1161.772886] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1165.050589] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1167.624838] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1171.216665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1171.316555] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1171.412925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1174.546579] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1178.138627] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1178.182681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1181.963773] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1010h08m35s [ 1182.784677] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1186.663181] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1188.125023] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1188.196756] LustreError: 58092:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1208.477824] Lustre: lustre-MDT0000: Client 907a5766-ec96-4e44-9ec1-5181580771d6 (at 192.168.203.48@tcp) reconnecting [ 1228.216156] LustreError: 58092:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1229.436242] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1230.818856] Lustre: DEBUG MARKER: done [ 1233.892335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1233.897817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.906474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1234.401354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1234.408572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1234.417473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1234.421270] Lustre: Skipped 1 previous similar message [ 1239.523489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1239.994323] Lustre: server umount lustre-OST0000 complete [ 1244.644391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.647134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1244.653875] Lustre: Skipped 1 previous similar message [ 1244.657087] Lustre: Skipped 1 previous similar message [ 1248.295339] Lustre: server umount lustre-MDT0000 complete [ 1250.480360] LustreError: 57335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786370984 with bad export cookie 16045636747473091668 [ 1250.484728] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1250.491426] LustreError: 57335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1250.685809] Lustre: server umount lustre-MDT0001 complete [ 1256.207690] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 1257.726128] Key type lgssc unregistered [ 1257.890646] LNet: 61667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1257.895552] LNetError: 61667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1257.904376] LNet: Removed LNI 192.168.203.148@tcp [ 1258.402601] Key type .llcrypt unregistered [ 1258.404672] Key type ._llcrypt unregistered [ 1282.519648] Key type ._llcrypt registered [ 1282.520943] Key type .llcrypt registered [ 1282.567330] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 1283.046834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1283.096409] alg: No test for adler32 (adler32-zlib) [ 1284.042632] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 1284.185900] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 1285.792174] Key type lgssc registered [ 1286.388317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1288.104337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1288.113322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.256097] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1289.273686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1289.370938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.900808] Lustre: Failing over lustre-MDT0000 [ 1290.053441] Lustre: server umount lustre-MDT0000 complete [ 1293.509875] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:10:26 (1786371026) [ 1297.021637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1297.251949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1299.697711] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1304.561048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.627572] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1304.653788] Lustre: 64441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1304.825681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1307.441263] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1311.894084] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1315.564670] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1319.532384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1319.646046] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1319.768571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1323.148469] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1326.162615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1327.032625] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1330.707554] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1335.776951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1335.781603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.788418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1336.291638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1336.293544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.296511] Lustre: Skipped 1 previous similar message [ 1336.302051] Lustre: Skipped 1 previous similar message [ 1339.955350] Lustre: server umount lustre-MDT0000 complete [ 1341.811338] LustreError: 65175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371075 with bad export cookie 11751361576672548323 [ 1341.814898] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1341.825722] LustreError: 65175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1341.979353] Lustre: server umount lustre-MDT0001 complete [ 1343.895767] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1346.607548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1351.353310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.648955] LustreError: 63366:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001b36fa33 x1873145776604672/t0(0) o250->MGC192.168.203.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1351.829455] LustreError: 66289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.849906] LustreError: 66289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1351.898302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1351.970060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1354.078536] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1357.282895] LustreError: 66290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.284711] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.389172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1360.867936] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1363.424155] Lustre: 63369:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786371080/real 1786371080] req@00000000f7784e69 x1873145776604416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786371096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1363.943941] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1364.852902] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1367.014799] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1367.736095] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1368.544221] Lustre: 63370:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786371085/real 1786371085] req@0000000008625894 x1873145776604544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786371101 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1372.131107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1372.131762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1372.136946] Lustre: Skipped 2 previous similar messages [ 1372.143940] Lustre: Skipped 2 previous similar messages [ 1377.020334] Lustre: server umount lustre-MDT0000 complete [ 1377.250616] LustreError: 68935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1377.261444] LustreError: 68935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1378.983912] LustreError: 64433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371112 with bad export cookie 11751361576672550605 [ 1378.987986] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1378.989337] LustreError: 64433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1379.161661] Lustre: server umount lustre-MDT0001 complete [ 1382.970697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1388.213211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.512879] LustreError: 63366:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007a7398c9 x1873145776625664/t0(0) o250->MGC192.168.203.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1388.693300] LustreError: 66290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.746983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.752324] Lustre: Skipped 1 previous similar message [ 1388.793438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1391.030311] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1394.148026] LustreError: 68936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1394.148983] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1395.926974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1398.176315] Lustre: 63369:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786371116/real 1786371116] req@000000001975c13c x1873145776625280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786371132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1398.534414] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1401.327152] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1402.428498] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1403.362904] Lustre: 63368:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786371121/real 1786371121] req@00000000ffb1c08f x1873145776625536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786371137 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1403.381864] Lustre: 63368:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1405.443378] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1411.554991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1411.560068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1411.569621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1414.793922] Lustre: server umount lustre-MDT0000 complete [ 1416.712479] LustreError: 65175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371150 with bad export cookie 11751361576672551984 [ 1416.718275] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1416.719825] LustreError: 65175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1416.866635] Lustre: server umount lustre-MDT0001 complete [ 1420.455789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1422.641125] Lustre: DEBUG MARKER: files compared the same [ 1434.302429] Lustre: server umount lustre-OST0000 complete [ 1443.857648] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 1445.544327] Key type lgssc unregistered [ 1445.726585] LNet: 74158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1445.730878] LNetError: 74158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1445.741028] LNet: Removed LNI 192.168.203.148@tcp [ 1446.242080] Key type .llcrypt unregistered [ 1446.244719] Key type ._llcrypt unregistered [ 1461.336747] Key type ._llcrypt registered [ 1461.339361] Key type .llcrypt registered [ 1461.416178] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 1462.095367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1462.108501] alg: No test for adler32 (adler32-zlib) [ 1463.038966] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 1463.196887] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 1464.824167] Key type lgssc registered [ 1465.461717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1470.132514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1470.144473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1471.368422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.552863] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug +malloc all [ 1478.398034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.561602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1480.841097] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug +malloc all [ 1484.849963] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1487.915837] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1491.835801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1491.998900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1495.415463] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug +malloc all [ 1496.120111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1499.709474] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1511.392896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1511.397961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.407127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1513.443790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1513.443815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1513.451631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.690110] Lustre: server umount lustre-OST0000 complete [ 1518.560967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.561723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1518.568533] Lustre: Skipped 1 previous similar message [ 1518.574116] Lustre: Skipped 2 previous similar messages [ 1523.681313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.685115] Lustre: Skipped 1 previous similar message [ 1528.801480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1528.804667] Lustre: Skipped 1 previous similar message [ 1532.384126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1532.479708] Lustre: server umount lustre-MDT0000 complete [ 1533.920671] LustreError: 75554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1533.929423] LustreError: 75554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1534.500504] LustreError: 75533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371268 with bad export cookie 3438163133705862596 [ 1534.508044] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1534.509224] LustreError: 75533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1534.669740] Lustre: server umount lustre-MDT0001 complete [ 1540.871548] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 1542.409757] Key type lgssc unregistered [ 1542.601740] LNet: 79089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.608895] LNetError: 79089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1542.621660] LNet: Removed LNI 192.168.203.148@tcp [ 1543.131730] Key type .llcrypt unregistered [ 1543.133797] Key type ._llcrypt unregistered [ 1558.812857] Key type ._llcrypt registered [ 1558.815277] Key type .llcrypt registered [ 1558.906503] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 1559.582847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1559.644639] alg: No test for adler32 (adler32-zlib) [ 1560.571367] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 1560.699797] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 1562.312288] Key type lgssc registered [ 1562.980021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1567.430406] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1567.445717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1582.944390] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1588.064290] LustreError: 80457:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.148@tcp: failed processing log, type 4: rc = -110 [ 1614.688277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.210706] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1622.659721] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1626.867366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.945375] Lustre: *** cfs_fail_loc=706, val=0*** [ 1627.184658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.231477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1629.163565] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1633.202496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.377111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1635.365319] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1639.353717] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1642.889628] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1647.585344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1647.589958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.598396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1648.609193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1648.615472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.626655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1650.481537] Lustre: server umount lustre-OST0000 complete [ 1653.729235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1653.735610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1653.747647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1653.751291] Lustre: Skipped 2 previous similar messages [ 1659.020493] Lustre: server umount lustre-MDT0000 complete [ 1659.368281] LustreError: 81544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1659.380334] LustreError: 81544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1661.182766] LustreError: 80465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371395 with bad export cookie 18413871497817052312 [ 1661.183484] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1661.192329] LustreError: 80465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1661.374098] Lustre: server umount lustre-MDT0001 complete [ 1667.224575] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 1668.751349] Key type lgssc unregistered [ 1668.940822] LNet: 84072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.945701] LNetError: 84072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.958691] LNet: Removed LNI 192.168.203.148@tcp [ 1669.356172] Key type .llcrypt unregistered [ 1669.358274] Key type ._llcrypt unregistered [ 1682.416989] Key type ._llcrypt registered [ 1682.418520] Key type .llcrypt registered [ 1682.488426] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 1683.020213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1683.099274] alg: No test for adler32 (adler32-zlib) [ 1684.024410] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 1684.162079] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 1685.776159] Key type lgssc registered [ 1686.574370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1691.031351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1691.041059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1694.350268] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1698.479581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.782137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1700.889761] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1703.905932] LustreError: 86151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1703.917020] LustreError: 86151:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1705.529672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1705.692786] LustreError: 86154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1705.702209] LustreError: 86154:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1705.708595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1708.608147] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1708.772346] LustreError: 86155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1708.785446] LustreError: 86155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1712.862330] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1712.870457] Lustre: MGS: Client 308b4e9e-adf2-42d0-903d-6fda45842a3b (at 0@lo) reconnecting [ 1712.882098] LustreError: 86165:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1713.010462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1713.053422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1715.044221] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1728.483535] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1728.484517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1728.491426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1728.497082] Lustre: Skipped 1 previous similar message [ 1728.501367] Lustre: Skipped 1 previous similar message [ 1731.301202] Lustre: server umount lustre-OST0000 complete [ 1733.601351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1733.601740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.602310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1733.612908] Lustre: Skipped 1 previous similar message [ 1738.720925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1738.725984] Lustre: Skipped 2 previous similar messages [ 1739.352800] Lustre: server umount lustre-MDT0000 complete [ 1741.402420] Lustre: server umount lustre-MDT0001 complete [ 1743.194692] Lustre: server umount MGS complete [ 1748.320261] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:18:01 (1786371481) [ 1761.762039] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 1766.613378] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 1784.241860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.516493] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1791.841099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.960404] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1791.987514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1792.039805] Lustre: lustre-MDT0001: new disk, initializing [ 1792.086795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1792.102274] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1792.106222] Lustre: 93311:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1792.110963] LustreError: 93311:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1794.217550] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1800.240614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1800.394252] Lustre: lustre-OST0000: new disk, initializing [ 1800.397138] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1800.401574] Lustre: 94077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1800.449568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.911213] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1809.256913] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1809.954619] Lustre: 94743:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000f2903567 x1873146196038016/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786371559 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1812.953378] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1812.960913] Lustre: MGS: Client 52086f05-ec9a-4fe8-bca5-bcad4bbd889a (at 0@lo) reconnecting [ 1812.982187] Lustre: 92581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1812.991665] LustreError: 93297:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1812.997681] LustreError: 93297:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1813.010462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1813.050553] Lustre: lustre-MDT0000: new disk, initializing [ 1813.137110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1813.157793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1814.177129] Lustre: 95143:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d17b46cf x1873146196043520/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786371564 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1814.305223] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1815.385332] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1816.231400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1816.239035] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1816.370286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1836.004835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1836.006195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1836.013454] Lustre: Skipped 1 previous similar message [ 1836.018264] Lustre: Skipped 1 previous similar message [ 1841.259455] Lustre: server umount lustre-OST0000 complete [ 1844.196362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1844.197790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1844.206958] Lustre: Skipped 1 previous similar message [ 1844.220263] Lustre: Skipped 3 previous similar messages [ 1849.364621] Lustre: server umount lustre-MDT0000 complete [ 1857.563542] Lustre: server umount lustre-MDT0001 complete [ 1859.575471] Lustre: server umount MGS complete [ 1865.239552] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:19:58 (1786371598) [ 1873.792873] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 1875.190916] Key type lgssc unregistered [ 1875.361082] LNet: 97642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1875.371995] LNetError: 97642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1875.383612] LNet: Removed LNI 192.168.203.148@tcp [ 1875.855760] Key type .llcrypt unregistered [ 1875.857385] Key type ._llcrypt unregistered [ 1885.285245] Key type ._llcrypt registered [ 1885.286932] Key type .llcrypt registered [ 1885.341041] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 1885.863787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1885.871548] alg: No test for adler32 (adler32-zlib) [ 1886.787802] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 1886.921615] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 1888.536620] Key type lgssc registered [ 1889.058502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1892.844671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1892.856157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.085191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1896.038646] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1900.232615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.422765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1902.667968] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1907.985914] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1913.079289] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1919.171969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1919.544131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.994504] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 1931.504827] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1939.147616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1949.669225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.671366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1949.684409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1954.802828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1954.806376] Lustre: Skipped 1 previous similar message [ 1955.211701] Lustre: server umount lustre-OST0000 complete [ 1959.910478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1959.916775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.926488] Lustre: Skipped 2 previous similar messages [ 1959.944543] Lustre: Skipped 1 previous similar message [ 1965.027268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1965.033237] Lustre: Skipped 1 previous similar message [ 1970.146288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.153121] Lustre: Skipped 1 previous similar message [ 1973.216197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1973.431627] Lustre: server umount lustre-MDT0000 complete [ 1975.272318] LustreError: 100454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1975.285274] LustreError: 100454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1976.222730] LustreError: 98823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371710 with bad export cookie 944970068202708325 [ 1976.230985] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1976.441544] Lustre: server umount lustre-MDT0001 complete [ 1984.234701] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 1986.160651] Key type lgssc unregistered [ 1986.367039] LNet: 102398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.371528] LNetError: 102398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1986.397066] LNet: Removed LNI 192.168.203.148@tcp [ 1987.108528] Key type .llcrypt unregistered [ 1987.110578] Key type ._llcrypt unregistered [ 1999.315154] Key type ._llcrypt registered [ 1999.317651] Key type .llcrypt registered [ 1999.401680] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2000.369594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2000.450602] alg: No test for adler32 (adler32-zlib) [ 2001.534723] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 2001.769964] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 2003.448235] Key type lgssc registered [ 2004.364820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2009.026667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2009.026736] LustreError: 103461:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2009.032981] Lustre: Skipped 1 previous similar message [ 2009.040327] LustreError: 103460:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2009.046375] LustreError: 103461:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 2009.053391] LustreError: 103461:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2009.068027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.068255] LustreError: 103461:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2010.397884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2016.846856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.022358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2020.376799] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2024.271277] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing lsmod [ 2025.920781] LustreError: 104651:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2026.003408] LustreError: 104666:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2026.008940] LustreError: 104651:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4918 [ 2026.015294] LustreError: 104651:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2026.022513] LustreError: 104651:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2026.024767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2026.211776] LustreError: 104677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.225848] LustreError: 104677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2026.300655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2031.586674] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.589782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.608434] Lustre: Skipped 1 previous similar message [ 2031.624943] Lustre: Skipped 2 previous similar messages [ 2034.312114] Lustre: server umount lustre-MDT0000 complete [ 2036.707303] LustreError: 103498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.723158] LustreError: 103498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2036.745827] LustreError: 103474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371770 with bad export cookie 5439441788630621146 [ 2036.755989] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2036.756082] LustreError: 103474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2037.792972] LustreError: 103930:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003884bb67 x1873146528418816/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2037.903154] Lustre: server umount lustre-MDT0001 complete [ 2050.144511] Lustre: server umount lustre-OST0000 complete [ 2057.287759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.600191] LustreError: 105732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2057.660744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2060.803230] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2062.820079] LustreError: 105733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.817078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.089509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2070.883614] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2076.438317] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2080.468208] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2085.367606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2085.567796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.260859] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2095.335788] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2103.083442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2113.508852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2113.509182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.516528] LustreError: Skipped 1 previous similar message [ 2113.525074] Lustre: Skipped 1 previous similar message [ 2113.530032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2113.541857] Lustre: Skipped 1 previous similar message [ 2116.380172] Lustre: server umount lustre-OST0000 complete [ 2123.746915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.751843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.763172] Lustre: Skipped 2 previous similar messages [ 2125.223593] Lustre: server umount lustre-MDT0000 complete [ 2127.980557] LustreError: 105712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371861 with bad export cookie 5439441788630621846 [ 2127.993114] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.259749] Lustre: server umount lustre-MDT0001 complete [ 2136.069232] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 2137.938868] Key type lgssc unregistered [ 2138.154422] LNet: 109288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.163367] LNetError: 109288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.182542] LNet: Removed LNI 192.168.203.148@tcp [ 2138.776335] Key type .llcrypt unregistered [ 2138.778099] Key type ._llcrypt unregistered [ 2155.897456] Key type ._llcrypt registered [ 2155.899249] Key type .llcrypt registered [ 2155.981922] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2156.565719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2156.596768] alg: No test for adler32 (adler32-zlib) [ 2157.581729] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 2157.763249] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 2159.400161] Key type lgssc registered [ 2160.352802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2165.693456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2165.715069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2166.980894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.850594] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2175.425740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2175.663236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2178.846586] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2183.456544] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2187.495803] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2192.436741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2192.686038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2197.009470] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2202.049831] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2211.185140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2217.159358] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2219.349783] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2223.858958] Lustre: server umount lustre-OST0000 complete [ 2226.658283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.666779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2226.677431] Lustre: Skipped 1 previous similar message [ 2231.777602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.786186] Lustre: Skipped 2 previous similar messages [ 2236.898788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.910509] Lustre: Skipped 1 previous similar message [ 2236.954911] Lustre: server umount lustre-MDT0000 complete [ 2239.512181] LustreError: 110665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786371973 with bad export cookie 5995799877243757101 [ 2239.523695] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.524788] LustreError: 110665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2249.796444] Lustre: server umount lustre-MDT0001 complete [ 2257.216576] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 2259.268361] Key type lgssc unregistered [ 2259.477753] LNet: 114342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2259.482577] LNetError: 114342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2259.493671] LNet: Removed LNI 192.168.203.148@tcp [ 2260.088090] Key type .llcrypt unregistered [ 2260.090404] Key type ._llcrypt unregistered [ 2273.053729] Key type ._llcrypt registered [ 2273.056063] Key type .llcrypt registered [ 2273.140777] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2273.836546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2273.861217] alg: No test for adler32 (adler32-zlib) [ 2274.887544] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 2275.079327] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 2276.696174] Key type lgssc registered [ 2277.424179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2282.281536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2282.293807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2283.514641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.102385] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2291.806661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2292.043823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2294.640737] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2299.101694] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2302.130871] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2306.139648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2306.367061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.399903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2309.803784] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2314.404183] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2323.938495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2323.943826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.951370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2327.010801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2327.010862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2327.014570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2327.032984] Lustre: Skipped 1 previous similar message [ 2329.334301] Lustre: server umount lustre-OST0000 complete [ 2332.132198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.133670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2332.141410] Lustre: Skipped 1 previous similar message [ 2337.250442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.256682] Lustre: Skipped 2 previous similar messages [ 2337.615737] Lustre: server umount lustre-MDT0000 complete [ 2339.721341] LustreError: 115522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786372073 with bad export cookie 1329893544929857173 [ 2339.723696] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2339.733820] LustreError: 115522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2339.892628] Lustre: server umount lustre-MDT0001 complete [ 2346.793603] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 2348.673174] Key type lgssc unregistered [ 2348.933881] LNet: 119074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2348.938766] LNetError: 119074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2348.952661] LNet: Removed LNI 192.168.203.148@tcp [ 2349.542430] Key type .llcrypt unregistered [ 2349.546986] Key type ._llcrypt unregistered [ 2365.466069] Key type ._llcrypt registered [ 2365.467943] Key type .llcrypt registered [ 2365.556333] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2366.430127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.453440] alg: No test for adler32 (adler32-zlib) [ 2367.427289] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 2367.592898] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 2369.232668] Key type lgssc registered [ 2370.153267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2375.158323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2375.173163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.509715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2379.740852] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2385.973930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.223921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2389.467354] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2394.503485] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2398.189637] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2402.836647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2403.076987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.270451] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2412.131477] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2422.517443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2427.339612] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2429.325686] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2429.329179] Lustre: Skipped 2 previous similar messages [ 2431.539551] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2431.543481] Lustre: Skipped 2 previous similar messages [ 2437.148794] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2437.167143] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2440.228778] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.48@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2440.239532] Lustre: Skipped 2 previous similar messages [ 2445.319225] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.48@tcp 192.168.0.[2,10]@tcp [ 2448.009404] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.48/24@tcp in log lustre-MDT0000 [ 2448.017412] Lustre: Skipped 2 previous similar messages [ 2451.846785] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.48/24@tcp [ 2451.850775] Lustre: Skipped 1 previous similar message [ 2458.595128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.598141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2458.603974] Lustre: Skipped 1 previous similar message [ 2458.609700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2463.714592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2463.717936] Lustre: Skipped 2 previous similar messages [ 2464.485232] Lustre: server umount lustre-OST0000 complete [ 2468.321263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2468.325168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.333814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2472.456931] Lustre: server umount lustre-MDT0000 complete [ 2473.953288] LustreError: 122909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2473.960063] LustreError: 122909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2474.324856] LustreError: 120452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786372208 with bad export cookie 5854963066059973839 [ 2474.327237] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2474.330376] LustreError: 120452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2474.538947] Lustre: server umount lustre-MDT0001 complete [ 2479.487414] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 2481.044945] Key type lgssc unregistered [ 2481.208890] LNet: 125574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2481.213495] LNetError: 125574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2481.223588] LNet: Removed LNI 192.168.203.148@tcp [ 2481.646085] Key type .llcrypt unregistered [ 2481.648137] Key type ._llcrypt unregistered [ 2496.123154] Key type ._llcrypt registered [ 2496.124767] Key type .llcrypt registered [ 2496.176746] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2496.634254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2496.687099] alg: No test for adler32 (adler32-zlib) [ 2497.605153] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 2497.739103] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 2499.345146] Key type lgssc registered [ 2499.913666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2503.386801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2503.396729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.578472] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2504.593516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2504.598240] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.48/24@tcp [ 2506.573627] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2510.581911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.750705] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2510.767213] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.48/24@tcp [ 2510.784405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2512.839238] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2516.398357] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2518.885928] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2522.072308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2522.234367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.819593] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2527.884291] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2529.388603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2545.122489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2545.128451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2545.137296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2546.145562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2546.145726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2546.152897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.932341] Lustre: server umount lustre-OST0000 complete [ 2551.266342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.267020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2551.274328] Lustre: Skipped 1 previous similar message [ 2551.280647] Lustre: Skipped 2 previous similar messages [ 2555.771302] Lustre: server umount lustre-MDT0000 complete [ 2556.385634] LustreError: 127111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.394822] LustreError: 127111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2557.580601] LustreError: 127093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786372291 with bad export cookie 11036172090042340096 [ 2557.587289] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2557.735427] Lustre: server umount lustre-MDT0001 complete [ 2562.553073] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 2563.821944] Key type lgssc unregistered [ 2563.972809] LNet: 130721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2563.976364] LNetError: 130721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2563.986333] LNet: Removed LNI 192.168.203.148@tcp [ 2564.354498] Key type .llcrypt unregistered [ 2564.356320] Key type ._llcrypt unregistered [ 2575.935863] Key type ._llcrypt registered [ 2575.937529] Key type .llcrypt registered [ 2575.987896] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 2583.609427] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2584.150234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2584.178050] alg: No test for adler32 (adler32-zlib) [ 2585.093763] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 2585.225436] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 2586.816307] Key type lgssc registered [ 2587.409737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2603.545906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2603.560357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.664536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2604.680603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2604.721259] Lustre: lustre-MDT0000: new disk, initializing [ 2604.757784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2604.770679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2606.619944] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2611.758607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.809661] Lustre: 134097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2611.838463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2611.842534] Lustre: Skipped 1 previous similar message [ 2611.881739] Lustre: lustre-MDT0001: new disk, initializing [ 2611.920251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2611.936679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2611.942138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2613.699175] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2618.220258] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2620.695205] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2623.702100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2623.820478] Lustre: lustre-OST0000: new disk, initializing [ 2623.825422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2623.829275] Lustre: 136062:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2623.870024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2626.649677] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2631.253461] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2632.208970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2632.217481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2632.263014] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2633.283343] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2633.385532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2635.340588] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2635.455661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2637.281863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2637.281882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.286799] LustreError: Skipped 1 previous similar message [ 2637.287743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2637.293558] Lustre: Skipped 1 previous similar message [ 2637.303492] Lustre: Skipped 1 previous similar message [ 2638.499247] Lustre: server umount lustre-OST0000 complete [ 2642.401925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2642.402675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.408044] Lustre: Skipped 1 previous similar message [ 2646.504504] Lustre: server umount lustre-MDT0000 complete [ 2648.064974] LustreError: 136061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786372382 with bad export cookie 18220387497194394826 [ 2648.066553] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2648.070463] LustreError: 136061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2648.209756] Lustre: server umount lustre-MDT0001 complete [ 2651.710864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.938254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2653.667246] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2657.249427] LustreError: 138274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2657.259161] LustreError: 138274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2657.351348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.513401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2659.159786] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2662.026853] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2664.339782] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2667.163278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2667.317307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2669.731191] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2672.769474] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2674.579549] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2676.726431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2678.409086] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2678.506774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2682.263882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2682.342582] Lustre: lustre-OST0001: new disk, initializing [ 2682.345107] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2682.349690] Lustre: 141658:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2682.353163] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 2682.355093] Lustre: Skipped 1 previous similar message [ 2684.646398] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2687.469325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2687.475227] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2687.496937] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2688.404793] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2688.487364] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2690.189449] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2690.285121] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2692.555773] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95d208021000.ost_server_uuid 50 [ 2693.177410] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95d208021000.ost_server_uuid in FULL state after 0 sec [ 2697.702367] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2697.703970] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.712057] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2697.714681] Lustre: Skipped 1 previous similar message [ 2701.706715] Lustre: server umount lustre-OST0001 complete [ 2702.820613] LustreError: 140094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.827960] LustreError: 140094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2704.353266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2704.358211] LustreError: Skipped 1 previous similar message [ 2704.361157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.368375] Lustre: Skipped 1 previous similar message [ 2704.371648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2707.425993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2707.940285] LustreError: 140094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.947933] LustreError: 140094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2709.476972] Lustre: server umount lustre-OST0000 complete [ 2713.057947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2713.063549] Lustre: Skipped 1 previous similar message [ 2713.065918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2713.068518] Lustre: Skipped 3 previous similar messages [ 2725.344111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2725.428341] Lustre: server umount lustre-MDT0000 complete [ 2727.064391] LustreError: 141655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786372461 with bad export cookie 18220387497194395694 [ 2727.067897] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2727.070277] LustreError: 141655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2727.225126] Lustre: server umount lustre-MDT0001 complete [ 2734.696783] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 2735.881034] Key type lgssc unregistered [ 2736.015591] LNet: 144659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2736.019449] LNetError: 144659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2736.031409] LNet: Removed LNI 192.168.203.148@tcp [ 2736.451870] Key type .llcrypt unregistered [ 2736.453298] Key type ._llcrypt unregistered [ 2751.925700] Key type ._llcrypt registered [ 2751.927247] Key type .llcrypt registered [ 2751.974365] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 2758.900492] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2759.270904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2759.304068] alg: No test for adler32 (adler32-zlib) [ 2760.179356] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 2760.286726] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 2761.880205] Key type lgssc registered [ 2762.299958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2776.050287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2776.057789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2777.154811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2777.171579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2777.211335] Lustre: lustre-MDT0000: new disk, initializing [ 2777.245484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2777.255130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2778.789023] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2783.602567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2783.657863] Lustre: 148787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2783.687893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2783.691445] Lustre: Skipped 1 previous similar message [ 2783.728959] Lustre: lustre-MDT0001: new disk, initializing [ 2783.769678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2783.787179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2783.791888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2785.289111] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2789.358089] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2791.489846] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2793.976206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2794.090240] Lustre: lustre-OST0000: new disk, initializing [ 2794.092352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2794.095374] Lustre: 150755:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2794.133794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2796.309083] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2800.180504] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2800.629962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2800.633601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2800.664199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2807.260481] Lustre: Failing over lustre-OST0000 [ 2807.335186] Lustre: server umount lustre-OST0000 complete [ 2820.589060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2820.718614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2820.723781] Lustre: Skipped 1 previous similar message [ 2820.726893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.737216] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2821.779175] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2823.006633] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2824.511486] Lustre: Failing over lustre-MDT0000 [ 2824.608770] Lustre: server umount lustre-MDT0000 complete [ 2824.674078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2824.677979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.210798] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2827.232232] Lustre: 147015:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786372544/real 1786372544] req@0000000069762dc4 x1873147324297472/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786372560 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2831.328135] Lustre: 147014:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786372549/real 1786372549] req@00000000bd20f09e x1873147324299136/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786372565 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2831.331838] LustreError: 148794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.346595] LustreError: 148794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2832.004099] LustreError: 150650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.449541] LustreError: 148797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.456763] LustreError: 148797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2838.333765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.394159] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2838.399140] Lustre: 148789:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.148@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2838.488681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2838.514958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.544658] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2840.226965] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2842.243882] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2843.628043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2843.637108] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2843.666713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2848.739176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2848.742073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.743465] LustreError: Skipped 1 previous similar message [ 2848.748728] Lustre: Skipped 1 previous similar message [ 2848.751468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2848.757971] Lustre: Skipped 1 previous similar message [ 2851.095563] Lustre: server umount lustre-OST0000 complete [ 2853.858155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.858645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2853.864801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2853.869154] Lustre: Skipped 1 previous similar message [ 2858.670956] Lustre: server umount lustre-MDT0000 complete [ 2858.977444] LustreError: 148798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2858.984649] LustreError: 148798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2860.220508] LustreError: 148777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786372594 with bad export cookie 5621422953300765781 [ 2860.225450] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2860.228031] LustreError: 148777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2860.366376] Lustre: server umount lustre-MDT0001 complete [ 2864.328190] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 2865.366597] Key type lgssc unregistered [ 2865.494661] LNet: 154403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2865.497924] LNetError: 154403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2865.505303] LNet: Removed LNI 192.168.203.148@tcp [ 2865.815576] Key type .llcrypt unregistered [ 2865.817250] Key type ._llcrypt unregistered [ 2886.794263] Key type ._llcrypt registered [ 2886.795748] Key type .llcrypt registered [ 2886.838828] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2887.214136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2887.262194] alg: No test for adler32 (adler32-zlib) [ 2888.140260] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 2888.234097] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 2889.824238] Key type lgssc registered [ 2890.250454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2893.407508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2893.416149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.587909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.133893] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2900.435563] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2902.649657] LustreError: 155962:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2902.654224] LustreError: 155962:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2902.658508] LustreError: 155962:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 2902.835749] Lustre: server umount lustre-MDT0000 complete [ 2909.318738] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 2913.018287] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 2927.175588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.264392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2927.276967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2927.312790] Lustre: lustre-MDT0000: new disk, initializing [ 2927.334904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.343555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2928.790362] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2933.250650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.297531] Lustre: 159258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2933.325924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2933.328157] Lustre: Skipped 1 previous similar message [ 2933.363799] Lustre: lustre-MDT0001: new disk, initializing [ 2933.400492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2933.414090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2933.417958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2934.835244] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2938.812933] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2940.787715] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2943.148215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2943.250114] Lustre: lustre-OST0000: new disk, initializing [ 2943.252841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2943.255729] Lustre: 161225:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2943.286468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2945.260950] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 2949.244325] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2950.658391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2950.662732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2950.700794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2956.657934] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2957.239064] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:38:10 (1786372690) [ 3046.370186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3046.373700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.379631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3047.904991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.905056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3047.917281] Lustre: Skipped 2 previous similar messages [ 3049.317899] Lustre: server umount lustre-MDT0000 complete [ 3050.854920] LustreError: 159249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786372784 with bad export cookie 11372203720900206058 [ 3050.855987] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3050.859129] LustreError: 159249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3053.024892] LustreError: 162989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.025526] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3053.032046] LustreError: 162989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3053.037322] Lustre: Skipped 1 previous similar message [ 3053.042027] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3055.072473] LustreError: 159265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3055.072569] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3060.192458] LustreError: 161943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.192541] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3065.312129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3065.312416] LustreError: 159263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.383616] Lustre: server umount lustre-MDT0001 complete [ 3081.184122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3081.243515] Lustre: server umount lustre-OST0000 complete [ 3084.814602] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 3089.022383] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 3104.529159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.628457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3104.640171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3104.679423] Lustre: lustre-MDT0000: new disk, initializing [ 3104.712434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3104.721199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3106.135474] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3111.085902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.131890] Lustre: 165858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3111.160986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3111.165478] Lustre: Skipped 1 previous similar message [ 3111.206286] Lustre: lustre-MDT0001: new disk, initializing [ 3111.265973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3111.270939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3113.206441] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3117.497327] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3119.465842] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3122.328380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.446744] Lustre: lustre-OST0000: new disk, initializing [ 3122.449435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3122.453469] Lustre: 167825:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3122.495262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3122.498377] Lustre: Skipped 1 previous similar message [ 3124.693628] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3128.923891] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3129.852696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3129.856946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3129.894666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3130.505293] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3130.585409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3132.369059] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3132.470493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3134.947044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3134.947587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.949309] LustreError: Skipped 1 previous similar message [ 3134.953758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3134.956652] Lustre: Skipped 2 previous similar messages [ 3139.363207] Lustre: server umount lustre-OST0000 complete [ 3142.113399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3142.116578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.121790] Lustre: Skipped 1 previous similar message [ 3146.920917] Lustre: server umount lustre-MDT0000 complete [ 3148.250199] LustreError: 166652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786372882 with bad export cookie 11372203720900364874 [ 3148.252943] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3148.254871] LustreError: 166652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3148.363122] Lustre: server umount lustre-MDT0001 complete [ 3151.473973] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:41:24 (1786372884) [ 3161.110399] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 3169.223920] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 3193.020931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.175389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3193.192474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3193.251338] Lustre: lustre-MDT0000: new disk, initializing [ 3193.298334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3193.315461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3196.913607] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3203.978781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3204.063768] Lustre: 172575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3204.250775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3207.013231] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3212.326207] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3214.937514] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3218.345620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3218.509126] Lustre: 174551:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3221.531183] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3226.964841] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3228.171445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3228.179986] Lustre: Skipped 1 previous similar message [ 3228.183349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3228.235927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3238.374502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3238.374781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.375918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3238.375923] Lustre: Skipped 3 previous similar messages [ 3238.380014] LustreError: Skipped 1 previous similar message [ 3238.395665] Lustre: Skipped 2 previous similar messages [ 3241.380569] Lustre: server umount lustre-OST0000 complete [ 3249.409279] Lustre: server umount lustre-MDT0000 complete [ 3251.242364] LustreError: 173375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786372985 with bad export cookie 11372203720900365735 [ 3251.249780] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3251.399373] Lustre: server umount lustre-MDT0001 complete [ 3255.733694] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:43:09 (1786372989) [ 3264.710160] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 3270.147213] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 3288.567852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.749692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3288.772652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3288.777097] Lustre: Skipped 3 previous similar messages [ 3288.829754] Lustre: lustre-MDT0000: new disk, initializing [ 3288.832244] Lustre: Skipped 2 previous similar messages [ 3288.879792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3288.884443] Lustre: Skipped 2 previous similar messages [ 3291.121707] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3297.080336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.148108] Lustre: 178866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3297.298614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3297.304499] Lustre: Skipped 1 previous similar message [ 3297.309900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3299.704604] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3305.337224] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3308.407639] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3312.088448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3312.253258] Lustre: 180839:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3313.051353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3315.803408] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3321.555889] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3328.483755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3328.484226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.490225] LustreError: Skipped 1 previous similar message [ 3328.497834] Lustre: Skipped 2 previous similar messages [ 3328.501691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3328.508342] Lustre: Skipped 6 previous similar messages [ 3330.934367] Lustre: server umount lustre-OST0000 complete [ 3338.907599] Lustre: server umount lustre-MDT0000 complete [ 3340.437681] LustreError: 178859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373074 with bad export cookie 11372203720900366827 [ 3340.438435] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3340.444460] LustreError: 178859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3340.568280] Lustre: server umount lustre-MDT0001 complete [ 3344.909147] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 3346.177676] Key type lgssc unregistered [ 3346.327339] LNet: 182695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3346.332488] LNetError: 182695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3346.342646] LNet: Removed LNI 192.168.203.148@tcp [ 3346.724272] Key type .llcrypt unregistered [ 3346.725715] Key type ._llcrypt unregistered [ 3358.023328] Key type ._llcrypt registered [ 3358.024593] Key type .llcrypt registered [ 3358.084601] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 3358.546622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3358.586066] alg: No test for adler32 (adler32-zlib) [ 3359.489042] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 3359.597200] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 3361.208130] Key type lgssc registered [ 3361.738186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3365.614068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3365.624021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3366.845858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3369.078800] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3373.223736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3373.412493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3375.336836] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3378.756066] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3381.355478] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3384.477888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3384.637912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3387.263819] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3390.582240] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3391.850923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3417.570701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3417.576079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.583689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3419.620563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.621268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3419.621344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3420.648429] Lustre: server umount lustre-OST0000 complete [ 3424.738589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.739333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3424.740653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3424.740659] Lustre: Skipped 1 previous similar message [ 3424.745165] Lustre: Skipped 1 previous similar message [ 3428.614371] Lustre: server umount lustre-MDT0000 complete [ 3429.858538] LustreError: 184816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.870203] LustreError: 184816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3430.546605] LustreError: 184070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373164 with bad export cookie 7399055776430006605 [ 3430.549383] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3430.554632] LustreError: 184070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3430.736535] Lustre: server umount lustre-MDT0001 complete [ 3435.644166] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 3436.994816] Key type lgssc unregistered [ 3437.140503] LNet: 187641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3437.145372] LNetError: 187641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3437.159606] LNet: Removed LNI 192.168.203.148@tcp [ 3437.595705] Key type .llcrypt unregistered [ 3437.597468] Key type ._llcrypt unregistered [ 3449.095637] Key type ._llcrypt registered [ 3449.096779] Key type .llcrypt registered [ 3449.146154] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 3449.665916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3449.688539] alg: No test for adler32 (adler32-zlib) [ 3450.567822] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 3450.674254] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 3452.272157] Key type lgssc registered [ 3452.701870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3456.071165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3456.079312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3457.276371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3459.076788] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3463.051049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.235609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3465.148614] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3468.485497] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3470.933205] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3473.817632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3473.968655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3476.481843] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3479.965460] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3482.154619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3493.346791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3493.349540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3493.349981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3493.352759] Lustre: Skipped 1 previous similar message [ 3494.629535] Lustre: server umount lustre-OST0000 complete [ 3497.822416] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff95d21a29f800.ost_server_uuid 50 [ 3513.214394] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3524.068182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.069987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3524.073362] Lustre: Skipped 1 previous similar message [ 3524.075382] Lustre: Skipped 1 previous similar message [ 3529.185757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3529.190882] Lustre: Skipped 2 previous similar messages [ 3530.160240] Lustre: server umount lustre-MDT0000 complete [ 3531.842174] LustreError: 189016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373265 with bad export cookie 4236520912046355599 [ 3531.845286] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3531.847947] LustreError: 189016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3532.008695] Lustre: server umount lustre-MDT0001 complete [ 3535.307612] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:47:48 (1786373268) [ 3538.197638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3538.408748] LustreError: 192978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.419356] LustreError: 192978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3538.441606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3540.371087] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3543.521647] LustreError: 192979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3544.435123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.615867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3546.576427] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3550.079706] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3552.766307] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3556.141572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3556.288248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3558.726092] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3561.846709] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3563.500374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3564.566287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3564.650452] Lustre: lustre-OST0001: new disk, initializing [ 3564.652973] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3564.657638] Lustre: 195817:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3567.148950] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3571.213869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3571.219193] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3571.255498] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3571.695317] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3573.729099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3573.734492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3573.743651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3575.266244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3578.819557] Lustre: server umount lustre-OST0000 complete [ 3581.416760] LustreError: 194801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3581.423449] LustreError: 194801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3581.601695] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3581.712744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3583.426112] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3583.510436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3586.530510] LustreError: 194799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3586.538417] LustreError: 194799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3591.651914] LustreError: 194801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3591.662751] LustreError: 194801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3596.773092] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3596.773375] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.774648] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3596.774653] Lustre: Skipped 4 previous similar messages [ 3596.777310] LustreError: Skipped 1 previous similar message [ 3596.782735] Lustre: Skipped 2 previous similar messages [ 3600.100114] Lustre: server umount lustre-OST0001 complete [ 3601.890130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3601.896271] Lustre: Skipped 1 previous similar message [ 3607.009395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3607.015175] Lustre: Skipped 4 previous similar messages [ 3616.224354] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3616.302106] Lustre: server umount lustre-MDT0000 complete [ 3617.961535] LustreError: 192960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373351 with bad export cookie 4236520912046356649 [ 3617.965464] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3617.975985] LustreError: 192960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3618.141916] Lustre: server umount lustre-MDT0001 complete [ 3629.140933] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:49:22 (1786373362) [ 3632.287565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3632.434380] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3632.454671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3632.544295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.547732] Lustre: Skipped 1 previous similar message [ 3634.474475] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3638.374356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3638.431588] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3638.452112] Lustre: 199241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3640.436428] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3643.845612] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3646.444869] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3649.943544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3650.037760] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3651.575091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3653.016173] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3656.304541] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3659.643221] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3659.682575] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3660.483435] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3662.783740] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3666.172190] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3671.013662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3671.019945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3671.022803] Lustre: Skipped 5 previous similar messages [ 3673.486103] Lustre: server umount lustre-OST0000 complete [ 3676.136084] LustreError: 202211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.146949] LustreError: 202211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3686.370248] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3686.375322] LustreError: Skipped 1 previous similar message [ 3690.463133] Lustre: server umount lustre-OST0001 complete [ 3695.073039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3695.077735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3695.084381] Lustre: Skipped 3 previous similar messages [ 3706.336109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3706.849311] LustreError: 199253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3706.856375] LustreError: 199253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3708.469557] Lustre: server umount lustre-MDT0000 complete [ 3709.923184] LustreError: 199234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373443 with bad export cookie 4236520912046358042 [ 3709.925841] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3709.928324] LustreError: 199234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3710.069114] Lustre: server umount lustre-MDT0001 complete [ 3719.665829] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:50:53 (1786373453) [ 3725.860898] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 3729.345723] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 3744.677769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3744.776152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3744.794494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3744.835943] Lustre: lustre-MDT0000: new disk, initializing [ 3744.867354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3744.871303] Lustre: Skipped 3 previous similar messages [ 3744.878633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3746.676249] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3751.975046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.028370] Lustre: 207306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3752.060907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3752.063904] Lustre: Skipped 1 previous similar message [ 3752.103555] Lustre: lustre-MDT0001: new disk, initializing [ 3752.175206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3752.180993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3753.941441] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3758.170892] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3760.240388] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3762.579631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3762.679795] Lustre: lustre-OST0000: new disk, initializing [ 3762.682066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3762.685313] Lustre: 209273:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3764.756404] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3768.755545] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3768.829888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3768.834030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3768.869625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3770.300172] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3770.380365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3771.767172] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3771.842980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3773.925172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3773.925512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3773.926978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3773.926985] Lustre: Skipped 8 previous similar messages [ 3773.937935] Lustre: Skipped 2 previous similar messages [ 3778.651799] Lustre: server umount lustre-OST0000 complete [ 3786.405110] Lustre: server umount lustre-MDT0000 complete [ 3787.986368] LustreError: 207298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373521 with bad export cookie 4236520912046359813 [ 3787.989799] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3787.992210] LustreError: 207298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3788.135481] Lustre: server umount lustre-MDT0001 complete [ 3792.110823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3792.315351] LustreError: 211487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3794.209156] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3797.791947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.717874] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3802.806166] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3805.136607] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3807.664408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3810.047848] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3812.881267] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3814.506583] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3818.675930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3820.100370] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3822.205697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 3823.072817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3823.075213] LustreError: Skipped 2 previous similar messages [ 3828.946474] Lustre: server umount lustre-OST0000 complete [ 3830.994573] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3831.076037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3832.472740] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3856.110261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3856.218299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3856.221609] Lustre: Skipped 5 previous similar messages [ 3858.542766] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3862.361481] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3863.378870] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3868.001141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3868.003535] Lustre: Skipped 7 previous similar messages [ 3872.100604] Lustre: server umount lustre-OST0000 complete [ 3874.796347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3874.803139] Lustre: Skipped 4 previous similar messages [ 3880.659451] Lustre: server umount lustre-MDT0000 complete [ 3883.491299] LustreError: 213947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3883.513325] LustreError: 213947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3883.633415] LustreError: 211467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373617 with bad export cookie 4236520912046360681 [ 3883.635081] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3883.639755] LustreError: 211467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3883.858883] Lustre: server umount lustre-MDT0001 complete [ 3889.553499] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:53:42 (1786373622) [ 3893.324913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.560913] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3901.633454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3904.359554] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3908.109369] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3911.054779] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3914.453228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3917.597731] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3921.166667] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3923.352907] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3923.462565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3925.756263] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3928.950803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 3932.112412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3932.230271] Lustre: lustre-OST0001: new disk, initializing [ 3932.233707] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3932.240325] Lustre: 220894:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3935.098914] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3937.815631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3937.821297] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3937.873516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3939.957385] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3942.089467] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3942.186301] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3944.175575] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3944.304220] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3948.006822] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3948.011681] LustreError: Skipped 3 previous similar messages [ 3951.309473] Lustre: server umount lustre-OST0001 complete [ 3954.277709] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3954.410850] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3956.526121] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3956.657589] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3981.523828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3984.715898] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 3986.921155] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3986.926839] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3986.926949] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3986.937263] Lustre: Skipped 1 previous similar message [ 3988.273417] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3989.083207] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3997.156604] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3997.159658] Lustre: Skipped 11 previous similar messages [ 4004.320111] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4004.377690] Lustre: server umount lustre-OST0001 complete [ 4012.520218] LustreError: 219327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4012.527506] LustreError: 219327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 4017.634823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.646836] Lustre: Skipped 6 previous similar messages [ 4023.779223] Lustre: server umount lustre-OST0000 complete [ 4031.776655] Lustre: server umount lustre-MDT0000 complete [ 4033.537539] LustreError: 217488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373767 with bad export cookie 4236520912046361619 [ 4033.539601] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4033.544479] LustreError: 217488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4039.768374] Lustre: server umount lustre-MDT0001 complete [ 4050.698606] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:56:24 (1786373784) [ 4053.660754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4053.788121] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4053.804509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4053.900786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4053.903981] Lustre: Skipped 5 previous similar messages [ 4055.761643] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4059.220454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.279460] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4059.296366] Lustre: 226126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4061.294805] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4064.394222] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4066.754700] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4069.519406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4069.606452] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4071.981141] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4075.117803] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4079.265226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4079.306526] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4081.583699] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4084.367472] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4085.012833] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4086.795383] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4086.893681] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4088.764460] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4088.866737] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4091.167204] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95d204ae8000.ost_server_uuid 50 [ 4091.732206] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95d204ae8000.ost_server_uuid in FULL state after 0 sec [ 4092.092488] Lustre: Permanently deactivating lustre-OST0001 [ 4092.099870] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4093.063934] Lustre: Permanently reactivating lustre-OST0001 [ 4098.021520] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4098.025545] LustreError: Skipped 3 previous similar messages [ 4100.220961] Lustre: server umount lustre-OST0001 complete [ 4107.677335] Lustre: server umount lustre-OST0000 complete [ 4115.177976] Lustre: server umount lustre-MDT0000 complete [ 4116.635518] LustreError: 229000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373850 with bad export cookie 4236520912046362893 [ 4116.635829] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4116.640537] LustreError: 229000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4116.775213] Lustre: server umount lustre-MDT0001 complete [ 4125.852187] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:57:39 (1786373859) [ 4128.517561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4128.635689] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4128.648191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4128.652287] Lustre: Skipped 5 previous similar messages [ 4130.148185] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4133.301957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.347932] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4133.360844] Lustre: 232806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4134.923548] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4137.521505] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4139.653929] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4142.329663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4142.418897] Lustre: Permanently deactivating lustre-OST0000 [ 4142.424486] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4144.490623] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4147.181448] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4148.723068] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4148.726548] LustreError: 188345:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004144d05c x1873148048141056/t0(0) o8->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4149.570314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4151.604331] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4154.064811] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4154.917629] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4160.039082] Lustre: Permanently reactivating lustre-OST0000 [ 4160.045217] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4160.047947] Lustre: Skipped 2 previous similar messages [ 4161.287134] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4161.294692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4169.924097] Lustre: server umount lustre-OST0001 complete [ 4177.232765] Lustre: server umount lustre-OST0000 complete [ 4180.571351] Lustre: server umount lustre-MDT0000 complete [ 4181.713118] LustreError: 232799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373915 with bad export cookie 4236520912046364510 [ 4181.716475] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4181.717055] LustreError: 232799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4181.825922] Lustre: server umount lustre-MDT0001 complete [ 4185.113448] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 4186.066617] Key type lgssc unregistered [ 4186.188383] LNet: 237658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4186.190814] LNetError: 237658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4186.203273] LNet: Removed LNI 192.168.203.148@tcp [ 4186.483552] Key type .llcrypt unregistered [ 4186.484728] Key type ._llcrypt unregistered [ 4194.656915] Key type ._llcrypt registered [ 4194.658167] Key type .llcrypt registered [ 4194.696806] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 4195.070525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4195.094364] alg: No test for adler32 (adler32-zlib) [ 4195.947801] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 4196.033812] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 4197.616219] Key type lgssc registered [ 4197.976069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4201.327171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4201.337336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4202.493662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4203.880530] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4206.851360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4206.894937] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4206.904686] Lustre: 239091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4206.909332] Lustre: Permanently deactivating lustre-MDT0001 [ 4206.914044] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4207.027112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4208.109613] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4208.112438] LustreError: 239135:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4208.117105] LustreError: 239135:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4208.121397] LustreError: 239135:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 4208.389883] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4210.772723] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4212.612864] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4214.836360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4214.950414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4216.855127] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4216.995826] LustreError: 240941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4216.998899] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4217.003299] LustreError: 240941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4219.201862] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4221.397656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4221.464797] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4223.338333] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4225.868089] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4226.537881] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4227.561196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4227.561562] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4231.720715] Lustre: Permanently reactivating lustre-MDT0001 [ 4231.725746] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4231.728031] Lustre: Skipped 2 previous similar messages [ 4232.968865] Lustre: lustre-MDT0001: Client 9e2e9213-cb9b-4c04-a5e3-a7241f861482 (at 192.168.203.48@tcp) reconnecting [ 4235.702332] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4236.351752] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4238.284270] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4238.356943] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4240.201399] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4240.313099] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4241.717337] Lustre: Permanently deactivating lustre-MDT0001 [ 4241.729501] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4241.735855] Lustre: Skipped 2 previous similar messages [ 4247.572177] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4253.883665] Lustre: server umount lustre-MDT0000 complete [ 4257.257804] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.261838] LustreError: 239098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4257.275307] Lustre: Skipped 2 previous similar messages [ 4257.300171] LustreError: 239098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4258.289939] LustreError: 239130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786373992 with bad export cookie 10850899949308897471 [ 4258.296220] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4258.302081] LustreError: 239130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4260.555495] Lustre: server umount lustre-MDT0001 complete [ 4265.393284] Lustre: server umount lustre-OST0000 complete [ 4277.116548] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:00:10 (1786374010) [ 4288.493923] Lustre: server umount lustre-OST0001 complete [ 4296.961712] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 4305.752242] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 4341.312433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.431606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4341.437889] Lustre: Skipped 2 previous similar messages [ 4341.452250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4341.504273] Lustre: lustre-MDT0000: new disk, initializing [ 4341.558696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.582826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4345.518899] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4354.896352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4355.039173] Lustre: 247357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4355.103399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4355.109175] Lustre: Skipped 1 previous similar message [ 4355.185252] Lustre: lustre-MDT0001: new disk, initializing [ 4355.275167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4355.323133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4355.329137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4359.920308] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4368.774911] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4374.070245] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4379.201903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4379.460318] Lustre: lustre-OST0000: new disk, initializing [ 4379.465455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4379.471407] Lustre: 249326:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4379.550507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4384.110376] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4385.308929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4385.317310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4385.485096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4391.803486] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4393.439306] LustreError: 248450:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4400.799639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4400.926162] Lustre: lustre-OST0001: new disk, initializing [ 4400.929415] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4400.934718] Lustre: 250439:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4403.473106] LustreError: 248450:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4403.575796] LustreError: 250114:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4406.070907] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4408.916271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4408.926591] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4409.050495] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4413.672083] LustreError: 250114:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4414.093451] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4415.174184] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4419.556092] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4419.560702] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4419.562128] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4419.565315] LustreError: Skipped 1 previous similar message [ 4419.574819] Lustre: Skipped 2 previous similar messages [ 4423.787107] Lustre: server umount lustre-OST0001 complete [ 4424.699211] LustreError: 249312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4424.752988] LustreError: 249312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4429.802725] LustreError: 249758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4429.836185] LustreError: 249758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4434.917506] LustreError: 249311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.942770] LustreError: 249311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4440.034639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.049197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4440.055020] Lustre: Skipped 2 previous similar messages [ 4445.155887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4445.164608] LustreError: 251411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4445.168954] Lustre: Skipped 1 previous similar message [ 4445.181910] LustreError: 251411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4445.415904] Lustre: server umount lustre-OST0000 complete [ 4450.276336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.280126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4450.290047] Lustre: Skipped 2 previous similar messages [ 4450.297956] Lustre: Skipped 1 previous similar message [ 4454.619735] Lustre: server umount lustre-MDT0000 complete [ 4457.896380] LustreError: 247351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374191 with bad export cookie 10850899949308900285 [ 4457.911034] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4457.933784] LustreError: 247351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4472.288159] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4472.752365] Lustre: server umount lustre-MDT0001 complete [ 4482.015472] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 4484.218230] Key type lgssc unregistered [ 4484.437856] LNet: 252475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4484.447937] LNetError: 252475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4484.463723] LNet: Removed LNI 192.168.203.148@tcp [ 4485.209806] Key type .llcrypt unregistered [ 4485.214655] Key type ._llcrypt unregistered [ 4515.156950] Key type ._llcrypt registered [ 4515.159616] Key type .llcrypt registered [ 4515.235858] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 4516.345628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4516.387988] alg: No test for adler32 (adler32-zlib) [ 4517.410264] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 4517.588620] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 4519.256237] Key type lgssc registered [ 4520.067684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4526.283668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4526.296524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4527.468323] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4527.483583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4527.591709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4531.302945] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4537.313965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4537.377651] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4537.400834] Lustre: 254663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4537.581943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4540.973551] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4546.335826] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4550.227483] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4554.854655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4554.971481] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4555.092257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4559.255129] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4560.977350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4563.833444] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4612.065750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4612.074142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4612.099611] LustreError: Skipped 1 previous similar message [ 4612.131244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4612.141176] Lustre: Skipped 1 previous similar message [ 4613.908921] Lustre: server umount lustre-OST0000 complete [ 4617.309281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4626.691902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4626.915965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4631.725651] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4632.039648] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4632.045310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4632.047568] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4632.106823] Lustre: Skipped 1 previous similar message [ 4637.640814] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4678.120435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4678.127318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4678.148647] Lustre: Skipped 1 previous similar message [ 4678.153322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4680.982171] Lustre: server umount lustre-OST0000 complete [ 4686.306026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4686.310731] LustreError: Skipped 1 previous similar message [ 4686.313495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4686.321018] Lustre: Skipped 1 previous similar message [ 4686.323916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4686.327369] Lustre: Skipped 1 previous similar message [ 4688.353529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.354122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4688.394577] Lustre: Skipped 1 previous similar message [ 4689.704829] Lustre: server umount lustre-MDT0000 complete [ 4692.112845] LustreError: 255413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374426 with bad export cookie 14977034866015538157 [ 4692.118531] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4692.127498] LustreError: 255413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4692.322286] Lustre: server umount lustre-MDT0001 complete [ 4699.421398] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 4701.356331] Key type lgssc unregistered [ 4701.570478] LNet: 260080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4701.574854] LNetError: 260080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4701.584643] LNet: Removed LNI 192.168.203.148@tcp [ 4702.159437] Key type .llcrypt unregistered [ 4702.162667] Key type ._llcrypt unregistered [ 4719.008625] Key type ._llcrypt registered [ 4719.012434] Key type .llcrypt registered [ 4719.114952] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 4719.879621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4719.915118] alg: No test for adler32 (adler32-zlib) [ 4720.934962] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 4721.122920] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 4722.776102] Key type lgssc registered [ 4723.538608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4728.650543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4728.664221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4729.904943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4732.578402] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4738.086772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4738.306618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4741.617236] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4746.208795] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4749.389736] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4753.144689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4753.340238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4754.358088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4756.847801] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4761.620513] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4778.980847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4778.981665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4778.990194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4779.003854] Lustre: Skipped 1 previous similar message [ 4783.162210] Lustre: server umount lustre-OST0000 complete [ 4789.218865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4789.220239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4789.225121] Lustre: Skipped 1 previous similar message [ 4789.229344] Lustre: Skipped 1 previous similar message [ 4791.657168] Lustre: server umount lustre-MDT0000 complete [ 4793.235726] LustreError: 261455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374527 with bad export cookie 11700516881235648688 [ 4793.239559] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4793.241143] LustreError: 261455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4793.416373] Lustre: server umount lustre-MDT0001 complete [ 4798.045091] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 4799.344724] Key type lgssc unregistered [ 4799.488576] LNet: 265780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4799.492178] LNetError: 265780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4799.501475] LNet: Removed LNI 192.168.203.148@tcp [ 4799.867901] Key type .llcrypt unregistered [ 4799.869538] Key type ._llcrypt unregistered [ 4808.072986] Key type ._llcrypt registered [ 4808.074717] Key type .llcrypt registered [ 4808.125338] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 4808.593888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4808.602794] alg: No test for adler32 (adler32-zlib) [ 4809.497959] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 4809.614816] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 4811.216189] Key type lgssc registered [ 4811.699927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4814.931485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4814.938512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.109758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4817.769329] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4821.143106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4821.301308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4822.936466] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4826.000736] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4828.212061] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4830.914065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4831.050926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4833.280299] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4836.033977] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4841.318982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4846.562737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4846.564095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4846.564821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4846.575797] Lustre: Skipped 1 previous similar message [ 4847.326915] Lustre: server umount lustre-OST0000 complete [ 4850.985725] Lustre: server umount lustre-MDT0000 complete [ 4851.682066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4851.682726] LustreError: 266977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4851.688942] Lustre: Skipped 1 previous similar message [ 4851.698251] LustreError: 266977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4852.576264] LustreError: 267007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374586 with bad export cookie 4592869281199073347 [ 4852.582377] LustreError: 267007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4852.582489] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4852.736849] Lustre: server umount lustre-MDT0001 complete [ 4856.838093] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 4857.969911] Key type lgssc unregistered [ 4858.116902] LNet: 270659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4858.120314] LNetError: 270659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4858.128724] LNet: Removed LNI 192.168.203.148@tcp [ 4858.477313] Key type .llcrypt unregistered [ 4858.478940] Key type ._llcrypt unregistered [ 4868.459355] Key type ._llcrypt registered [ 4868.460491] Key type .llcrypt registered [ 4868.498338] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 4868.845099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4868.866028] alg: No test for adler32 (adler32-zlib) [ 4869.732538] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 4869.825658] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 4871.408155] Key type lgssc registered [ 4871.832388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4874.742604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4874.750791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4875.917492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4877.453378] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4880.534747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4880.663049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4882.138402] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4884.616669] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4886.441824] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4888.854779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4888.972186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4890.925471] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4893.499462] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4900.263817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4912.098404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4912.100914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4912.105586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4915.681984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4915.683332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4915.685315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4917.644099] Lustre: server umount lustre-OST0000 complete [ 4920.801651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.802699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4920.806446] Lustre: Skipped 1 previous similar message [ 4920.811200] Lustre: Skipped 2 previous similar messages [ 4925.012892] Lustre: server umount lustre-MDT0000 complete [ 4925.922225] LustreError: 272050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4925.929130] LustreError: 272050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4926.262308] LustreError: 272036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374660 with bad export cookie 6263092914531259445 [ 4926.265469] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4926.267835] LustreError: 272036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4926.387763] Lustre: server umount lustre-MDT0001 complete [ 4929.956188] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 4931.001848] Key type lgssc unregistered [ 4931.127584] LNet: 276378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4931.130627] LNetError: 276378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4931.137424] LNet: Removed LNI 192.168.203.148@tcp [ 4931.443384] Key type .llcrypt unregistered [ 4931.445172] Key type ._llcrypt unregistered [ 4941.340626] Key type ._llcrypt registered [ 4941.342657] Key type .llcrypt registered [ 4941.438991] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 4942.231814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4942.255545] alg: No test for adler32 (adler32-zlib) [ 4943.209573] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 4943.389108] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 4945.048165] Key type lgssc registered [ 4945.853663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4950.647787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4950.660610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.936391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4954.766267] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4961.148178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4961.254475] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4961.378650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4964.541596] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4969.867897] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4973.562719] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4978.899251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4979.200549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4982.914487] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 4987.135126] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4996.607335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5006.819196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5006.820317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5006.821315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5006.832169] Lustre: Skipped 1 previous similar message [ 5009.721855] Lustre: server umount lustre-OST0000 complete [ 5011.938941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5011.939491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5011.947635] Lustre: Skipped 1 previous similar message [ 5011.950966] Lustre: Skipped 1 previous similar message [ 5017.057508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5017.061996] Lustre: Skipped 1 previous similar message [ 5022.176961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5022.180185] Lustre: Skipped 1 previous similar message [ 5026.272063] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5026.348507] Lustre: server umount lustre-MDT0000 complete [ 5028.223387] LustreError: 280561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374762 with bad export cookie 17689392165137581606 [ 5028.225252] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5028.228941] LustreError: 280561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5028.425574] Lustre: server umount lustre-MDT0001 complete [ 5033.631687] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 5034.988259] Key type lgssc unregistered [ 5035.150653] LNet: 281317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5035.154403] LNetError: 281317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5035.162645] LNet: Removed LNI 192.168.203.148@tcp [ 5035.588652] Key type .llcrypt unregistered [ 5035.590685] Key type ._llcrypt unregistered [ 5040.705810] Key type ._llcrypt registered [ 5040.707843] Key type .llcrypt registered [ 5040.776639] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5048.978115] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 5056.683355] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 5057.060923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5057.082088] alg: No test for adler32 (adler32-zlib) [ 5058.008184] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 5058.166728] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 5059.792173] Key type lgssc registered [ 5060.330909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5075.476957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5075.487187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5076.584767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5076.598444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5076.642544] Lustre: lustre-MDT0000: new disk, initializing [ 5076.681957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5076.689950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5078.492298] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5083.908028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5083.980325] Lustre: 284840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5084.011900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5084.015018] Lustre: Skipped 1 previous similar message [ 5084.067277] Lustre: lustre-MDT0001: new disk, initializing [ 5084.140870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5084.164874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5084.169693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5086.336300] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5091.577643] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5094.346520] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5097.582075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5097.723845] Lustre: lustre-OST0000: new disk, initializing [ 5097.726791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5097.731583] Lustre: 286808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5097.775960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5100.844742] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5103.122342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5103.127905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5103.193364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5105.516307] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5107.158768] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5107.258530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5108.903986] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5108.993983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5113.316885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5113.318123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.320207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5113.323539] Lustre: Skipped 1 previous similar message [ 5113.326838] Lustre: Skipped 1 previous similar message [ 5118.435339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5118.437510] Lustre: Skipped 1 previous similar message [ 5123.554880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5123.557261] Lustre: Skipped 1 previous similar message [ 5124.064124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5124.122172] Lustre: server umount lustre-OST0000 complete [ 5128.674834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5128.679380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5128.682926] Lustre: Skipped 1 previous similar message [ 5128.688345] Lustre: Skipped 1 previous similar message [ 5131.758281] Lustre: server umount lustre-MDT0000 complete [ 5133.345318] LustreError: 286807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374867 with bad export cookie 2312144870459497388 [ 5133.348834] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5133.351330] LustreError: 286807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5133.490744] Lustre: server umount lustre-MDT0001 complete [ 5136.880153] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:14:30 (1786374870) [ 5139.540270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5139.765474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5141.517928] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5144.806028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5144.958336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5146.554939] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5149.542182] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5151.756985] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5154.496858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5154.639875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5157.025030] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5159.807662] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5175.267790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5175.268030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5175.270519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5175.270870] LustreError: Skipped 2 previous similar messages [ 5175.275155] Lustre: Skipped 1 previous similar message [ 5179.573160] Lustre: server umount lustre-OST0000 complete [ 5185.506549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5185.507885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5185.514486] Lustre: Skipped 1 previous similar message [ 5185.517246] Lustre: Skipped 1 previous similar message [ 5187.500483] Lustre: server umount lustre-MDT0000 complete [ 5189.189365] LustreError: 291720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786374923 with bad export cookie 2312144870459498270 [ 5189.194174] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5189.195367] LustreError: 291720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5189.354550] Lustre: server umount lustre-MDT0001 complete [ 5193.780874] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 5195.080701] Key type lgssc unregistered [ 5195.241660] LNet: 292773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5195.245983] LNetError: 292773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5195.259238] LNet: Removed LNI 192.168.203.148@tcp [ 5195.691454] Key type .llcrypt unregistered [ 5195.693132] Key type ._llcrypt unregistered [ 5205.973383] Key type ._llcrypt registered [ 5205.974796] Key type .llcrypt registered [ 5206.024352] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 5213.076939] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 5213.566835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5213.574780] alg: No test for adler32 (adler32-zlib) [ 5214.450087] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 5214.550924] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 5216.152340] Key type lgssc registered [ 5216.606904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5233.057587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5233.067128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5234.169348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5234.183659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5234.224274] Lustre: lustre-MDT0000: new disk, initializing [ 5234.254702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5234.261879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5235.810641] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5240.490338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.534724] Lustre: 296570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5240.563174] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5240.565680] Lustre: Skipped 1 previous similar message [ 5240.603332] Lustre: lustre-MDT0001: new disk, initializing [ 5240.647398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5240.661297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5240.667810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5242.294834] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5246.528487] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5248.701368] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5251.421967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5251.555619] Lustre: lustre-OST2710: new disk, initializing [ 5251.558685] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5251.562966] Lustre: 298538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5251.604957] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5254.047773] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5258.490681] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5258.762688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5258.767988] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5258.813345] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5261.288034] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5261.379786] Lustre: lustre-OST03e8: new disk, initializing [ 5261.382765] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5261.387283] Lustre: 299601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5261.423789] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5263.652198] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5265.928621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5265.936277] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5265.971525] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5266.477781] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5269.401422] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5269.488362] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5271.124839] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5271.216040] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5272.899374] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5272.992718] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5274.581081] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5274.671096] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5276.130091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.133290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5276.135055] Lustre: Skipped 3 previous similar messages [ 5276.139186] Lustre: Skipped 3 previous similar messages [ 5281.250889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5281.253949] Lustre: Skipped 3 previous similar messages [ 5281.953674] Lustre: server umount lustre-MDT0000 complete [ 5283.423195] LustreError: 296561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786375017 with bad export cookie 15969876538386804263 [ 5283.426278] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5283.428257] LustreError: 296561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5283.570229] Lustre: server umount lustre-MDT0001 complete [ 5295.630447] Lustre: server umount lustre-OST2710 complete [ 5307.494714] Lustre: server umount lustre-OST03e8 complete [ 5314.180304] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 5317.568874] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 5330.906576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5331.002603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5331.016151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5331.064538] Lustre: lustre-MDT0000: new disk, initializing [ 5331.096399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5331.106875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5332.811819] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5337.558075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5337.608565] Lustre: 304842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5337.739867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5339.445604] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5343.878200] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5346.028473] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5348.493414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5348.612190] Lustre: lustre-OST0000: new disk, initializing [ 5348.613719] Lustre: Skipped 1 previous similar message [ 5348.615297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5348.618351] Lustre: Skipped 2 previous similar messages [ 5348.621651] Lustre: 306810:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5348.654185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5348.657037] Lustre: Skipped 1 previous similar message [ 5350.792922] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5354.672152] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5356.035531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5356.039419] Lustre: Skipped 1 previous similar message [ 5356.041505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5356.076866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5356.195170] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5356.282946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5357.793608] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5357.880311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5361.125241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5361.125348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5361.127682] LustreError: Skipped 1 previous similar message [ 5361.133473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5361.136747] Lustre: Skipped 1 previous similar message [ 5364.687470] Lustre: server umount lustre-OST0000 complete [ 5366.241662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.242355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5366.246348] Lustre: Skipped 2 previous similar messages [ 5371.361296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5371.364243] Lustre: Skipped 2 previous similar messages [ 5372.056794] Lustre: server umount lustre-MDT0000 complete [ 5373.286084] LustreError: 304832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786375107 with bad export cookie 15969876538386805922 [ 5373.289429] LustreError: 304832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5373.289840] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5373.403423] Lustre: server umount lustre-MDT0001 complete [ 5376.208300] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:18:29 (1786375109) [ 5376.828053] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5377.449480] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:18:31 (1786375111) [ 5378.611855] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 5388.342067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.451151] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5388.460742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5388.533214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5390.124621] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5393.348731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.397741] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5393.411583] Lustre: 310656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5395.116844] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5411.301448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5411.429962] Lustre: Denying initial registration attempt from nid 192.168.203.148@tcp, specified as failover [ 5411.436209] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5411.443156] LustreError: 312113:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 5411.448449] LustreError: 312113:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5411.452398] LustreError: 312113:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5411.511293] Lustre: server umount lustre-OST0000 complete [ 5411.513297] LustreError: 312113:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5419.490088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5419.491283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5419.646295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5419.659939] Lustre: Skipped 1 previous similar message [ 5430.240928] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5430.377926] Lustre: server umount lustre-MDT0000 complete [ 5433.445201] LustreError: 311399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786375167 with bad export cookie 15969876538386806797 [ 5433.452883] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5433.459542] LustreError: 311399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5433.769700] Lustre: server umount lustre-MDT0001 complete [ 5444.055888] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 5446.096415] Key type lgssc unregistered [ 5446.331635] LNet: 313236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5446.346447] LNetError: 313236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5446.358942] LNet: Removed LNI 192.168.203.148@tcp [ 5447.078563] Key type .llcrypt unregistered [ 5447.080069] Key type ._llcrypt unregistered [ 5457.979774] Key type ._llcrypt registered [ 5457.981781] Key type .llcrypt registered [ 5458.072796] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 5459.042094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5459.061568] alg: No test for adler32 (adler32-zlib) [ 5460.037714] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 5460.221647] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 5461.872257] Key type lgssc registered [ 5462.957461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5500.802363] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 5505.950980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5505.965964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5507.126668] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5507.140609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5507.226785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5509.895291] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5515.137404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.190634] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5515.209202] Lustre: 315983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5515.381700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5518.601469] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5527.144532] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5534.592038] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5556.072971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5556.261162] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5556.459142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5563.786335] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5572.391723] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5578.213093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5578.221089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.237290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5578.246033] Lustre: Skipped 1 previous similar message [ 5580.990994] Lustre: server umount lustre-OST0000 complete [ 5586.917476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5586.941417] LustreError: Skipped 1 previous similar message [ 5586.950657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.958988] Lustre: Skipped 1 previous similar message [ 5586.999123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5588.456069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.464554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5593.572223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5593.577770] Lustre: Skipped 2 previous similar messages [ 5598.700222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5598.703818] Lustre: Skipped 1 previous similar message [ 5599.200227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5599.551692] Lustre: server umount lustre-MDT0000 complete [ 5603.763359] LustreError: 315976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786375337 with bad export cookie 9289879190834156011 [ 5603.771387] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5603.777529] LustreError: 315976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5604.158462] Lustre: server umount lustre-MDT0001 complete [ 5614.456768] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 5616.971842] Key type lgssc unregistered [ 5617.184864] LNet: 319559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5617.190885] LNetError: 319559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5617.200223] LNet: Removed LNI 192.168.203.148@tcp [ 5618.123830] Key type .llcrypt unregistered [ 5618.125635] Key type ._llcrypt unregistered [ 5644.663906] Key type ._llcrypt registered [ 5644.669596] Key type .llcrypt registered [ 5644.769815] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 5647.334690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5647.411213] alg: No test for adler32 (adler32-zlib) [ 5648.612561] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 5649.043413] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 5650.704211] Key type lgssc registered [ 5651.767835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5659.535684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5659.546775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.947653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5666.720987] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5670.455422] LustreError: 320976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5670.481369] LustreError: 320976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5675.510403] LustreError: 320977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5677.063485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5677.415991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5682.572859] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5690.588374] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5696.468402] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5706.347444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5706.665086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5713.141723] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5720.035373] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5740.516078] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5740.523132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5740.573901] Lustre: Skipped 2 previous similar messages [ 5740.584424] Lustre: Skipped 1 previous similar message [ 5742.238379] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.48@tcp (stopping) [ 5742.242202] Lustre: Skipped 1 previous similar message [ 5745.314806] Lustre: server umount lustre-MDT0000 complete [ 5745.636656] LustreError: 322176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.656955] LustreError: 322176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5749.133394] LustreError: 320958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786375483 with bad export cookie 683593749114430560 [ 5749.137795] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5749.160148] LustreError: 320958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5750.757198] LustreError: 320972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5750.758909] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5750.770507] LustreError: 320972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5750.814603] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5754.883088] LustreError: 320973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5754.883600] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5754.915150] LustreError: 320973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5754.937901] Lustre: Skipped 1 previous similar message [ 5755.782608] Lustre: server umount lustre-MDT0001 complete [ 5762.416852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5770.932392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5774.839237] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x97c9ce49aa4c235 [ 5774.845182] Lustre: MGC192.168.203.148@tcp: Connection restored to 0@lo (at 0@lo) [ 5775.404734] LustreError: 322811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.533327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5775.576131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 5780.095526] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5784.826807] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5789.670044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.079584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5794.698893] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5795.307885] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5802.155364] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5804.359692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5810.952338] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5813.005927] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5821.415395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5821.417529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5821.437746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5821.451810] Lustre: Skipped 1 previous similar message [ 5823.674234] Lustre: server umount lustre-OST0000 complete [ 5831.650408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5831.659952] Lustre: Skipped 2 previous similar messages [ 5831.660316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5831.664192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5831.670824] Lustre: Skipped 1 previous similar message [ 5842.400466] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5842.575268] Lustre: server umount lustre-MDT0000 complete [ 5842.917103] LustreError: 324594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.942058] LustreError: 324594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5846.250969] LustreError: 320958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786375580 with bad export cookie 683593749114438197 [ 5846.262710] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5846.278984] LustreError: 320958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5846.697745] Lustre: server umount lustre-MDT0001 complete [ 5857.884756] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 5861.914863] Key type lgssc unregistered [ 5862.150848] LNet: 327274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5862.159360] LNetError: 327274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5862.171136] LNet: Removed LNI 192.168.203.148@tcp [ 5863.070362] Key type .llcrypt unregistered [ 5863.072965] Key type ._llcrypt unregistered [ 5886.541665] Key type ._llcrypt registered [ 5886.545657] Key type .llcrypt registered [ 5886.636384] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 5887.962803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5887.997056] alg: No test for adler32 (adler32-zlib) [ 5888.991294] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 5889.203440] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 5890.904170] Key type lgssc registered [ 5892.321236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5899.983901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5900.005036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.413980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5905.649835] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5911.142557] LustreError: 328692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5911.179074] LustreError: 328692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5915.003783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5915.284745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5920.527461] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5928.118685] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5932.719837] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5935.593100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5935.600428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5935.610399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5937.131124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5937.134584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5937.153058] Lustre: Skipped 1 previous similar message [ 5940.271323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5940.280396] Lustre: Skipped 1 previous similar message [ 5941.485643] Lustre: server umount lustre-MDT0000 complete [ 5945.320393] LustreError: 328693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5945.367070] LustreError: 328693:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5945.645749] LustreError: 328674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786375679 with bad export cookie 15354813497776967546 [ 5945.662657] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5945.998776] Lustre: server umount lustre-MDT0001 complete [ 5960.230658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5960.417612] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5960.438434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5960.606684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5965.383340] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5974.338056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5974.417905] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5974.451487] Lustre: 331275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5974.759251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5978.549375] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 5985.800657] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5990.978247] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5999.980058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6000.121560] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6000.132103] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6000.137829] LustreError: 333161:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6000.142425] LustreError: 333161:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6000.146572] LustreError: 333161:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6000.211113] Lustre: server umount lustre-OST0000 complete [ 6000.213410] LustreError: 333161:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6010.620756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6010.791944] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6010.987465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6016.577822] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6019.179244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 6023.353367] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6030.881688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6030.944887] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6031.004754] Lustre: lustre-OST0001: new disk, initializing [ 6031.007935] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6031.013713] Lustre: 334704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6031.223874] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6036.913068] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6037.622974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6037.638585] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6037.969647] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6044.943412] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6048.234136] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6048.250528] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6048.266923] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6048.281358] Lustre: Skipped 1 previous similar message [ 6048.311295] Lustre: Skipped 1 previous similar message [ 6053.349662] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6053.381022] Lustre: Skipped 1 previous similar message [ 6062.059717] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6062.155948] Lustre: server umount lustre-OST0001 complete [ 6063.593441] LustreError: 333628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6063.635975] LustreError: 333628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6065.633690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6065.637550] LustreError: Skipped 1 previous similar message [ 6065.640538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6065.650582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6065.654641] Lustre: Skipped 2 previous similar messages [ 6068.706516] LustreError: 333628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6068.721948] LustreError: 333628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6071.745098] Lustre: server umount lustre-OST0000 complete [ 6077.410144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6077.419235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6077.430497] Lustre: Skipped 1 previous similar message [ 6084.065092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6084.068857] Lustre: Skipped 6 previous similar messages [ 6089.184104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6089.330357] Lustre: server umount lustre-MDT0000 complete [ 6092.398327] LustreError: 332024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786375826 with bad export cookie 15354813497776968099 [ 6092.413154] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6092.416073] LustreError: 332024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6092.825784] Lustre: server umount lustre-MDT0001 complete [ 6101.566400] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 6103.805259] Key type lgssc unregistered [ 6104.012707] LNet: 336668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6104.017591] LNetError: 336668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6104.030402] LNet: Removed LNI 192.168.203.148@tcp [ 6104.756427] Key type .llcrypt unregistered [ 6104.758424] Key type ._llcrypt unregistered [ 6166.348721] Key type ._llcrypt registered [ 6166.350952] Key type .llcrypt registered [ 6166.428962] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 6178.330850] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 6179.353620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6179.395443] alg: No test for adler32 (adler32-zlib) [ 6180.396994] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 6180.560627] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 6182.184235] Key type lgssc registered [ 6183.006756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6215.418171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6215.434449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6216.570728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6216.592200] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6216.668296] Lustre: lustre-MDT0000: new disk, initializing [ 6216.754789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6216.770492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6220.160224] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6228.790976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6228.885417] Lustre: 341579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6228.965647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6228.971232] Lustre: Skipped 1 previous similar message [ 6229.050663] Lustre: lustre-MDT0001: new disk, initializing [ 6229.161369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6229.206312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6229.218892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6232.776134] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6240.606507] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6244.949711] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6250.425207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6250.640753] Lustre: lustre-OST0000: new disk, initializing [ 6250.644232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6250.651186] Lustre: 343548:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6250.742891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6255.615442] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6257.174402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6257.183537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6257.248394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6262.697461] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6266.565945] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6266.761539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6270.022225] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6270.182300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6272.484822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6272.486693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6272.494314] LustreError: Skipped 1 previous similar message [ 6272.501565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6272.506202] Lustre: Skipped 1 previous similar message [ 6277.722554] Lustre: server umount lustre-OST0000 complete [ 6280.675858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6280.680982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6280.688777] Lustre: Skipped 1 previous similar message [ 6280.693053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6282.722638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6282.723456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6286.861646] Lustre: server umount lustre-MDT0000 complete [ 6287.847165] LustreError: 342948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6287.866070] LustreError: 342948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6289.466884] LustreError: 341569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376023 with bad export cookie 15923510027502833290 [ 6289.473486] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6289.475030] LustreError: 341569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6289.708618] Lustre: server umount lustre-MDT0001 complete [ 6296.448520] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:33:49 (1786376029) [ 6302.316927] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:33:55 (1786376035) [ 6309.722069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6309.996223] LustreError: 346302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.053199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6313.339219] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6315.489970] LustreError: 346303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6315.504670] LustreError: 346303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6319.922237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6320.165167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6323.836067] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6329.314317] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6333.352254] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6338.114050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6338.376595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.643851] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6348.337877] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6365.186911] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6366.552467] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6367.798861] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6369.463760] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6370.884021] Lustre: Failing over lustre-MDT0000 [ 6371.066380] Lustre: server umount lustre-MDT0000 complete [ 6371.297135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6371.305624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.317806] LustreError: 347031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6375.603617] LustreError: 346732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6375.629201] LustreError: 346732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6385.816870] LustreError: 346297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6385.845213] LustreError: 346297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6387.666213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6387.741670] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6387.941636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6387.988506] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6390.833898] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6390.932879] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6393.324043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6393.351134] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6393.385479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6398.111649] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6399.166839] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6402.262350] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6403.816550] Lustre: Failing over lustre-MDT0000 [ 6404.028429] Lustre: server umount lustre-MDT0000 complete [ 6406.353132] LustreError: 346732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6406.361604] LustreError: 346732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6408.674719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6408.678031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6408.686493] Lustre: Skipped 3 previous similar messages [ 6420.874467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6420.950812] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6421.147327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6421.192714] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6421.644708] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6424.790693] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6426.603944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6426.612418] Lustre: Skipped 2 previous similar messages [ 6426.671064] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6426.696713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6432.815429] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6434.247292] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6441.954777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6441.959713] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.964357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6441.964365] Lustre: Skipped 1 previous similar message [ 6441.998996] Lustre: Skipped 3 previous similar messages [ 6445.013467] Lustre: server umount lustre-MDT0000 complete [ 6447.073483] LustreError: 347031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6447.087392] LustreError: 347031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6447.572563] LustreError: 346283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376181 with bad export cookie 15923510027502835999 [ 6447.589079] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6447.814372] Lustre: server umount lustre-MDT0001 complete [ 6461.230799] Lustre: server umount lustre-OST0000 complete [ 6472.910902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6476.617743] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6482.709427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6486.185928] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6491.727300] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6495.793918] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6500.614634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6500.826982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6500.833042] Lustre: Skipped 2 previous similar messages [ 6505.050462] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6510.652723] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6518.257512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6528.482917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6528.484621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6528.486871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6528.486882] Lustre: Skipped 2 previous similar messages [ 6528.489776] Lustre: Skipped 1 previous similar message [ 6528.512654] LustreError: Skipped 1 previous similar message [ 6530.559356] Lustre: server umount lustre-OST0000 complete [ 6538.723287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6538.728015] Lustre: Skipped 4 previous similar messages [ 6539.628395] Lustre: server umount lustre-MDT0000 complete [ 6542.311614] LustreError: 352635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376276 with bad export cookie 15923510027502836748 [ 6542.315144] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6542.317163] LustreError: 352635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6542.621963] Lustre: server umount lustre-MDT0001 complete [ 6550.098381] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 6551.958896] Key type lgssc unregistered [ 6552.181986] LNet: 356205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6552.188892] LNetError: 356205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6552.201665] LNet: Removed LNI 192.168.203.148@tcp [ 6552.798128] Key type .llcrypt unregistered [ 6552.805248] Key type ._llcrypt unregistered [ 6571.139991] Key type ._llcrypt registered [ 6571.144583] Key type .llcrypt registered [ 6571.272491] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 6572.282320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6572.301474] alg: No test for adler32 (adler32-zlib) [ 6573.293153] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 6573.485431] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 6575.128184] Key type lgssc registered [ 6576.618982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6582.663208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6582.677848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6583.966378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6586.702993] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6591.948082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6592.170822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6594.788589] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6599.883301] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6603.625938] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6608.672291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6608.918205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6613.399517] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6619.214649] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6628.396088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6638.562788] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6638.563376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6638.565576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6638.577395] Lustre: Skipped 2 previous similar messages [ 6643.154391] Lustre: server umount lustre-MDT0000 complete [ 6643.684362] LustreError: 357700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6643.700619] LustreError: 357700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6645.864704] LustreError: 357681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376379 with bad export cookie 7381597235330029200 [ 6645.870822] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6645.876316] LustreError: 357681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6646.065667] Lustre: server umount lustre-MDT0001 complete [ 6658.195959] Lustre: server umount lustre-OST0000 complete [ 6668.452567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6668.693830] LustreError: 361430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6668.745122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6671.864558] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6673.889557] LustreError: 361431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6677.898276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6678.115924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6680.906578] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6686.904756] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6691.044123] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6695.860758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6696.148096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6700.311225] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6705.463582] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6714.613798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6729.188365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6729.200747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6729.214903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6729.218793] Lustre: Skipped 2 previous similar messages [ 6730.213619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.215356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6730.225749] Lustre: Skipped 1 previous similar message [ 6730.249424] Lustre: Skipped 2 previous similar messages [ 6731.691244] Lustre: server umount lustre-MDT0000 complete [ 6734.165956] LustreError: 361410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376468 with bad export cookie 7381597235330030313 [ 6734.173976] LustreError: 361410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6734.174111] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6734.468856] Lustre: server umount lustre-MDT0001 complete [ 6747.074917] Lustre: server umount lustre-OST0000 complete [ 6757.557167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.811471] LustreError: 365361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6757.865038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6761.091532] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6762.980221] LustreError: 365362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6767.431284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6770.730747] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6775.477415] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6779.414456] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6784.503677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6788.606534] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6793.912406] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6802.096783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6808.546630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6808.557371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6808.574102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6809.898154] Lustre: server umount lustre-OST0000 complete [ 6813.666000] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6813.672055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6813.684630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6822.882846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6822.889467] Lustre: Skipped 3 previous similar messages [ 6826.976190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6827.091488] Lustre: server umount lustre-MDT0000 complete [ 6828.001577] LustreError: 366794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6828.011672] LustreError: 366794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6829.843211] LustreError: 365342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376563 with bad export cookie 7381597235330031335 [ 6829.846749] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6829.853359] LustreError: 365342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6830.112663] Lustre: server umount lustre-MDT0001 complete [ 6836.781663] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 6838.805736] Key type lgssc unregistered [ 6839.011511] LNet: 368917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6839.018397] LNetError: 368917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6839.035675] LNet: Removed LNI 192.168.203.148@tcp [ 6839.564375] Key type .llcrypt unregistered [ 6839.568748] Key type ._llcrypt unregistered [ 6857.443668] Key type ._llcrypt registered [ 6857.447861] Key type .llcrypt registered [ 6857.522709] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 6858.299992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6858.312443] alg: No test for adler32 (adler32-zlib) [ 6859.313748] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 6859.517177] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 6861.136235] Key type lgssc registered [ 6861.882486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6867.390377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6867.402975] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.409330] LustreError: 370336:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6867.423682] LustreError: 370336:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6867.430132] LustreError: 370336:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6867.435212] LustreError: 370336:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6867.440094] LustreError: 370336:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6872.362327] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6872.366160] LustreError: 370734:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6872.377633] LustreError: 370734:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6872.382238] LustreError: 370734:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6872.388488] LustreError: 370734:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6872.394326] LustreError: 370734:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6884.462195] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 6886.416487] Key type lgssc unregistered [ 6886.611404] LNet: 371738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6886.617395] LNetError: 371738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6886.632294] LNet: Removed LNI 192.168.203.148@tcp [ 6887.233506] Key type .llcrypt unregistered [ 6887.237697] Key type ._llcrypt unregistered [ 6898.414121] Key type ._llcrypt registered [ 6898.416780] Key type .llcrypt registered [ 6898.493852] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 6907.946988] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 6908.560569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6908.574843] alg: No test for adler32 (adler32-zlib) [ 6909.502105] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 6909.651607] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 6911.256157] Key type lgssc registered [ 6912.000743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6934.572315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6934.580779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6935.750974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6935.782231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6935.850303] Lustre: lustre-MDT0000: new disk, initializing [ 6935.915749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6935.928547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6938.834391] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6945.551743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6945.708774] Lustre: 374872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6945.750425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6945.753748] Lustre: Skipped 1 previous similar message [ 6945.798262] Lustre: lustre-MDT0001: new disk, initializing [ 6945.859591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6945.894530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6945.906150] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6948.472630] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6954.654933] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6958.954553] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6963.242464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6963.463488] Lustre: lustre-OST0000: new disk, initializing [ 6963.467285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6963.472900] Lustre: 376839:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6963.554439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6967.606760] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 6973.465956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6973.473059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6973.557226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6974.323116] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6977.302154] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6977.470772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6980.035283] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6980.216398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6983.651087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6983.658136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6983.662557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6983.673529] Lustre: Skipped 1 previous similar message [ 6987.502375] Lustre: server umount lustre-OST0000 complete [ 6991.842394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6991.850385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6991.859503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6991.863560] Lustre: Skipped 1 previous similar message [ 6993.890485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6993.891745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.921960] Lustre: server umount lustre-MDT0000 complete [ 6998.381875] LustreError: 374865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376732 with bad export cookie 5025681630303391941 [ 6998.388082] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6998.399990] LustreError: 374865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6998.619149] Lustre: server umount lustre-MDT0001 complete [ 7004.315187] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7005.363954] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:45:38 (1786376738) [ 7009.839140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7010.149661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7013.461576] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7015.394630] LustreError: 379294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.403771] LustreError: 379294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7019.581028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7019.794854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7022.942356] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7028.088013] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7031.923832] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7036.116574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7036.383865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7040.511354] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7045.538575] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7050.313873] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7050.407815] Lustre: lustre-OST0001: new disk, initializing [ 7050.410050] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7050.413467] Lustre: 382130:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7051.946688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7051.958254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7052.033742] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7054.371652] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7060.491376] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7065.574444] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7065.575408] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7065.578947] Lustre: Skipped 1 previous similar message [ 7065.586206] Lustre: Skipped 2 previous similar messages [ 7069.083601] Lustre: server umount lustre-OST0001 complete [ 7070.701137] LustreError: 382268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7070.715419] LustreError: 382268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7075.814929] LustreError: 381120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7075.824887] LustreError: 381120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7080.934645] LustreError: 381119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7080.940712] LustreError: 381119:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7085.537094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7085.544101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7085.554373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7085.558895] Lustre: Skipped 1 previous similar message [ 7086.054618] LustreError: 381120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7086.065207] LustreError: 381120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7096.292352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7096.294691] LustreError: 382268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7096.301166] Lustre: Skipped 5 previous similar messages [ 7096.315584] LustreError: 382268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7097.316233] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7097.454697] Lustre: server umount lustre-OST0000 complete [ 7101.410142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.417895] Lustre: Skipped 2 previous similar messages [ 7106.191766] Lustre: server umount lustre-MDT0000 complete [ 7108.523145] LustreError: 382129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376842 with bad export cookie 5025681630303392823 [ 7108.525628] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7108.530168] LustreError: 382129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7114.857478] Lustre: server umount lustre-MDT0001 complete [ 7122.547582] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 7124.269522] Key type lgssc unregistered [ 7124.456272] LNet: 384120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7124.462392] LNetError: 384120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7124.475822] LNet: Removed LNI 192.168.203.148@tcp [ 7125.049608] Key type .llcrypt unregistered [ 7125.051957] Key type ._llcrypt unregistered [ 7154.690461] Key type ._llcrypt registered [ 7154.692077] Key type .llcrypt registered [ 7154.784157] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 7155.481112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7155.505989] alg: No test for adler32 (adler32-zlib) [ 7156.469539] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 7156.633267] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 7158.256506] Key type lgssc registered [ 7159.049613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7164.577596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7164.598537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7165.742144] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7165.756741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7165.851841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7169.013987] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7174.375105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7174.440059] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7174.460385] Lustre: 386897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7174.632436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7177.481851] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7182.102032] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7185.591042] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7189.986593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7189.993188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7190.004228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7190.499629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7190.511828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7190.517334] Lustre: Skipped 1 previous similar message [ 7195.618837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7195.623062] Lustre: Skipped 1 previous similar message [ 7200.738186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7200.747183] Lustre: Skipped 1 previous similar message [ 7201.248123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7201.354103] Lustre: server umount lustre-MDT0000 complete [ 7203.426948] LustreError: 386889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786376937 with bad export cookie 14673456353699456146 [ 7203.433237] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7203.437073] LustreError: 386889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7203.637122] Lustre: server umount lustre-MDT0001 complete [ 7209.215850] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:49:02 (1786376942) [ 7212.897969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7213.224207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7216.093554] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7218.657598] LustreError: 389407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7218.668423] LustreError: 389407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7221.556039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7221.800753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7224.822418] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7229.964585] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7233.651427] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7237.804244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7237.914801] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7238.057778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7241.851729] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7246.393315] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7252.768229] Lustre: Permanently deactivating lustre-OST0000 [ 7252.779839] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7254.543517] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7255.591716] LustreError: 392249:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7255.600320] LustreError: 392249:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7255.605049] LustreError: 392249:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7257.308971] LustreError: 392297:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7257.312716] LustreError: 392297:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7264.875952] Lustre: server umount lustre-OST0000 complete [ 7266.670692] LustreError: 392546:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7266.676248] LustreError: 392546:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7272.419900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7272.422477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7272.427935] Lustre: Skipped 1 previous similar message [ 7278.613749] Lustre: server umount lustre-MDT0000 complete [ 7280.927390] LustreError: 391240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786377014 with bad export cookie 14673456353699456713 [ 7280.930168] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7280.933977] LustreError: 391240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7290.455541] Lustre: server umount lustre-MDT0001 complete [ 7295.534223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7298.200174] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7299.853423] LustreError: 393704:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7303.617697] Lustre: 394042:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.148@tcp [ 7304.940361] Lustre: server umount MGS complete [ 7312.007276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.246673] LustreError: 394766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7312.258828] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7312.263769] Lustre: Skipped 1 previous similar message [ 7312.304814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7314.785141] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7317.474861] LustreError: 394767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7320.068652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7320.212426] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7322.566435] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7327.441668] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7331.156611] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7332.378616] Lustre: Permanently reactivating lustre-OST0000 [ 7332.394903] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7332.402404] Lustre: Skipped 2 previous similar messages [ 7360.993782] LustreError: 396597:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7365.521977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7365.766208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7365.773211] Lustre: Skipped 1 previous similar message [ 7370.161981] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7375.955960] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7380.452181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7380.458168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7380.467392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7380.471497] Lustre: Skipped 3 previous similar messages [ 7381.473411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7385.079441] Lustre: server umount lustre-OST0000 complete [ 7391.713831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7391.722500] Lustre: Skipped 2 previous similar messages [ 7393.811355] Lustre: server umount lustre-MDT0000 complete [ 7396.059745] LustreError: 394748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786377130 with bad export cookie 14673456353699457749 [ 7396.060995] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7396.067976] LustreError: 394748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7396.257297] Lustre: server umount lustre-MDT0001 complete [ 7403.294025] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 7405.299366] Key type lgssc unregistered [ 7405.512878] LNet: 399323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7405.517588] LNetError: 399323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7406.567409] LNet: Removed LNI 192.168.203.148@tcp [ 7407.155703] Key type .llcrypt unregistered [ 7407.157751] Key type ._llcrypt unregistered [ 7420.070276] Key type ._llcrypt registered [ 7420.073215] Key type .llcrypt registered [ 7420.169975] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 7430.477669] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 7431.265688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7431.277127] alg: No test for adler32 (adler32-zlib) [ 7432.220827] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 7432.409191] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 7434.049441] Key type lgssc registered [ 7434.844583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7457.991220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7458.007716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.162691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7459.184546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7459.241719] Lustre: lustre-MDT0000: new disk, initializing [ 7459.297310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7459.311243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7461.503767] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7468.348272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7468.431095] Lustre: 402458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7468.520621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7468.524654] Lustre: Skipped 1 previous similar message [ 7468.630692] Lustre: lustre-MDT0001: new disk, initializing [ 7468.696808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7468.723203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7468.730897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7471.895798] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7478.479505] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7482.207186] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7486.679334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7486.858416] Lustre: lustre-OST0000: new disk, initializing [ 7486.861536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7486.866117] Lustre: 404424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7486.944602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7490.654791] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7492.632351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7492.637868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7492.707313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7496.833450] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7499.683246] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7499.830309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7502.672587] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7502.800189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7507.938967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7507.940931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7507.943868] LustreError: Skipped 1 previous similar message [ 7507.945301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7507.950923] Lustre: Skipped 1 previous similar message [ 7510.127144] Lustre: server umount lustre-OST0000 complete [ 7513.058159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.065264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7513.066665] Lustre: Skipped 1 previous similar message [ 7513.073014] Lustre: Skipped 2 previous similar messages [ 7518.178507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7518.182161] Lustre: Skipped 1 previous similar message [ 7523.300634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7523.309194] Lustre: Skipped 1 previous similar message [ 7526.368184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7526.476421] Lustre: server umount lustre-MDT0000 complete [ 7528.786617] LustreError: 402450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786377262 with bad export cookie 16924027159374076583 [ 7528.790072] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7528.796843] LustreError: 402450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7529.005575] Lustre: server umount lustre-MDT0001 complete [ 7534.458331] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7535.562809] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:54:28 (1786377268) [ 7539.977522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7540.267680] LustreError: 406881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7540.280763] LustreError: 406881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7540.311690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7543.533806] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7545.320326] LustreError: 406882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7548.944326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7549.176761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7552.218669] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7556.897913] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7560.434542] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7564.917388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7565.136841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7569.512661] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7573.986854] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7575.159483] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7578.196182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7581.677154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7581.691249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7581.702925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7584.743203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7584.746561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7587.642743] Lustre: server umount lustre-OST0000 complete [ 7589.859733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.868388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7589.870806] Lustre: Skipped 1 previous similar message [ 7589.878872] Lustre: Skipped 3 previous similar messages [ 7604.193353] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7604.313450] Lustre: server umount lustre-MDT0000 complete [ 7606.594450] LustreError: 406862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786377340 with bad export cookie 16924027159374077465 [ 7606.599566] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7606.600128] LustreError: 406862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7606.832783] Lustre: server umount lustre-MDT0001 complete [ 7613.213160] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 7614.976698] Key type lgssc unregistered [ 7615.153859] LNet: 410606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7615.160645] LNetError: 410606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7615.175435] LNet: Removed LNI 192.168.203.148@tcp [ 7615.711652] Key type .llcrypt unregistered [ 7615.713757] Key type ._llcrypt unregistered [ 7631.723524] Key type ._llcrypt registered [ 7631.726168] Key type .llcrypt registered [ 7631.803507] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 7632.100425] Key type .llcrypt unregistered [ 7632.102385] Key type ._llcrypt unregistered [ 7646.141289] Key type ._llcrypt registered [ 7646.143726] Key type .llcrypt registered [ 7646.244052] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 7646.959672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7646.983871] alg: No test for adler32 (adler32-zlib) [ 7648.042469] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 7648.261595] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 7649.944420] Key type lgssc registered [ 7651.158973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7657.013848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7657.024563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7658.274927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7661.385737] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7666.937469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7667.145200] LustreError: 412803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7667.161111] LustreError: 412803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7667.181708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7668.197486] LustreError: 412804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7668.206481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7668.216675] LustreError: 412804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7671.160802] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7673.317042] LustreError: 412800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7673.326215] LustreError: 412800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7676.512454] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7678.437116] LustreError: 412804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7678.454493] LustreError: 412804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7681.147403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7681.408758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7684.124654] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7693.793989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7693.800600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7693.808290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7696.868076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7696.870528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7696.873149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7699.683219] Lustre: server umount lustre-OST0000 complete [ 7707.106320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7707.109744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7707.112707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7707.114627] Lustre: Skipped 1 previous similar message [ 7707.116636] Lustre: Skipped 1 previous similar message [ 7708.361166] Lustre: server umount lustre-MDT0000 complete [ 7710.936857] LustreError: 412783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786377444 with bad export cookie 7553782991469319348 [ 7710.945673] LustreError: 412783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7710.947851] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7711.149777] Lustre: server umount lustre-MDT0001 complete [ 7718.237792] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 7720.282620] Key type lgssc unregistered [ 7720.489267] LNet: 415949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7720.495958] LNetError: 415949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7720.505790] LNet: Removed LNI 192.168.203.148@tcp [ 7721.099367] Key type .llcrypt unregistered [ 7721.105890] Key type ._llcrypt unregistered [ 7737.062169] Key type ._llcrypt registered [ 7737.063639] Key type .llcrypt registered [ 7737.135415] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 7737.838928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7737.850733] alg: No test for adler32 (adler32-zlib) [ 7738.869682] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 7739.014754] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 7740.640598] Key type lgssc registered [ 7741.457700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7747.058133] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7747.070692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7762.528395] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7767.648329] LustreError: 417318:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.148@tcp: failed processing log, type 4: rc = -110 [ 7794.336203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7797.830718] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7802.202601] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7806.305082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.594211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7806.638396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7808.854896] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7813.892940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7814.093568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7816.699510] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7821.231918] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7824.714636] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7829.475671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7829.476410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7829.491344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7829.495012] Lustre: Skipped 1 previous similar message [ 7834.597447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7834.606803] Lustre: Skipped 1 previous similar message [ 7835.501811] Lustre: server umount lustre-OST0000 complete [ 7839.203824] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7839.212305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7839.220775] Lustre: Skipped 1 previous similar message [ 7839.224512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7840.001600] Lustre: server umount lustre-MDT0000 complete [ 7842.187710] LustreError: 419096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786377576 with bad export cookie 337155147949814024 [ 7842.194057] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7842.195597] LustreError: 419096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7842.432881] Lustre: server umount lustre-MDT0001 complete [ 7848.909754] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 7850.650700] Key type lgssc unregistered [ 7850.842633] LNet: 420887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7850.849489] LNetError: 420887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7850.863900] LNet: Removed LNI 192.168.203.148@tcp [ 7851.496901] Key type .llcrypt unregistered [ 7851.503713] Key type ._llcrypt unregistered [ 7867.153477] Key type ._llcrypt registered [ 7867.155893] Key type .llcrypt registered [ 7867.241797] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 7867.902330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7867.945567] alg: No test for adler32 (adler32-zlib) [ 7868.881589] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 7869.042621] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 7870.704211] Key type lgssc registered [ 7871.620529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7876.875570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7876.888042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7878.118485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7880.371316] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7884.912336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7885.115975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7887.128815] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7891.876664] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7895.653592] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7899.848770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7900.134424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7903.219381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7904.981271] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 7912.080534] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7919.780729] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.48@tcp (stopping) [ 7920.626251] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7920.631066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7920.658377] Lustre: Skipped 2 previous similar messages [ 7920.677861] Lustre: Skipped 2 previous similar messages [ 7923.453237] Lustre: server umount lustre-MDT0000 complete [ 7924.890888] LustreError: 422275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7924.940747] LustreError: 422275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7925.734237] LustreError: 422281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7925.742754] LustreError: 422261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.48@tcp arrived at 1786377659 with bad export cookie 18323127900100284666 [ 7925.748921] LustreError: 422281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7925.816476] LustreError: 422261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7928.269412] LustreError: 422260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786377662 with bad export cookie 18323127900100283763 [ 7928.282552] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7930.850155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7930.852644] LustreError: 423341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7930.854105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7930.854982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7930.868112] LustreError: 423341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7933.859332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7933.861484] LustreError: 422280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.870986] LustreError: 422280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7934.443578] Lustre: server umount lustre-OST0000 complete [ 7938.979079] LustreError: 422281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7939.015568] LustreError: 422281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7945.897961] Lustre: server umount lustre-MDT0001 complete [ 7959.522805] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 7961.893599] Key type lgssc unregistered [ 7962.103356] LNet: 425975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7962.115433] LNetError: 425975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7962.167294] LNet: Removed LNI 192.168.203.148@tcp [ 7963.305281] Key type .llcrypt unregistered [ 7963.307736] Key type ._llcrypt unregistered [ 7993.013599] Key type ._llcrypt registered [ 7993.015544] Key type .llcrypt registered [ 7993.107360] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 7994.923235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7995.768315] alg: No test for adler32 (adler32-zlib) [ 7997.255432] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 7997.531808] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 7999.336179] Key type lgssc registered [ 8001.128218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8011.175460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8011.189086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8012.570985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8017.443903] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8023.011287] LustreError: 427399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8023.054922] LustreError: 427399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8026.852834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8027.244445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8032.377485] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8040.958354] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8046.213317] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8054.620085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8054.896938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8059.172975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8060.398562] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8068.471463] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8074.213277] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8074.244907] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8074.256589] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8076.421823] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.48@tcp (stopping) [ 8077.797549] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8077.840979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8077.866862] Lustre: Skipped 1 previous similar message [ 8079.567564] Lustre: server umount lustre-MDT0001 complete [ 8082.925394] LustreError: 427400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8082.947211] LustreError: 427400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8088.037634] LustreError: 427399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8088.041872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.090347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8091.506420] Lustre: server umount lustre-OST0000 complete [ 8093.158631] LustreError: 427400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8095.229338] Lustre: server umount lustre-MDT0000 complete [ 8108.242394] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 8110.849651] Key type lgssc unregistered [ 8111.075272] LNet: 431087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8111.088593] LNetError: 431087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8111.118415] LNet: Removed LNI 192.168.203.148@tcp [ 8112.397119] Key type .llcrypt unregistered [ 8112.399269] Key type ._llcrypt unregistered [ 8138.894960] Key type ._llcrypt registered [ 8138.898245] Key type .llcrypt registered [ 8138.997255] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 8153.366695] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 8155.769491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8155.811707] alg: No test for adler32 (adler32-zlib) [ 8156.935614] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 8157.349191] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 8159.078351] Key type lgssc registered [ 8160.818785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8199.513551] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 8216.511302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8216.611919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8217.836656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8217.872294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8217.923814] Lustre: lustre-MDT0000: new disk, initializing [ 8217.975705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8217.991502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8224.879295] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8241.252724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8241.513310] Lustre: lustre-OST0000: new disk, initializing [ 8241.517169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8241.521303] Lustre: Skipped 1 previous similar message [ 8241.528938] Lustre: 436121:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8241.594354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8249.370292] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8250.434495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8250.443879] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8250.561449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8259.397485] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8277.992155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.154389] Lustre: 435104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8278.317886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8278.505449] Lustre: lustre-MDT0001: new disk, initializing [ 8278.620391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8278.703600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8278.709640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8284.341216] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8295.346965] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8295.514541] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8300.669222] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8301.059525] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8308.197732] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8308.202085] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8308.204152] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8308.204804] Lustre: Skipped 1 previous similar message [ 8312.449676] Lustre: server umount lustre-MDT0001 complete [ 8318.436795] LustreError: 435114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8318.444863] LustreError: 435114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8323.567510] LustreError: 437826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8323.606346] LustreError: 437826:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8328.676464] LustreError: 438262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8328.687640] LustreError: 438262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8332.258096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8332.271330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8332.291647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8332.299819] Lustre: Skipped 2 previous similar messages [ 8333.796657] LustreError: 437826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8333.796683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8337.673245] Lustre: server umount lustre-OST0000 complete [ 8338.918826] LustreError: 435115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8356.321424] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8356.707335] Lustre: server umount lustre-MDT0000 complete [ 8373.315821] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 8375.300667] Key type lgssc unregistered [ 8375.544781] LNet: 439878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8375.553413] LNetError: 439878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8375.586959] LNet: Removed LNI 192.168.203.148@tcp [ 8376.465514] Key type .llcrypt unregistered [ 8376.470489] Key type ._llcrypt unregistered [ 8426.851375] Key type ._llcrypt registered [ 8426.853486] Key type .llcrypt registered [ 8427.009583] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 8427.562053] Key type .llcrypt unregistered [ 8427.564634] Key type ._llcrypt unregistered [ 8443.779681] Key type ._llcrypt registered [ 8443.781288] Key type .llcrypt registered [ 8443.860919] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 8446.071204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8446.271940] alg: No test for adler32 (adler32-zlib) [ 8447.820294] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 8448.395964] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 8450.160412] Key type lgssc registered [ 8451.555104] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8476.373268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8476.471270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8477.733818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8477.757191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8477.814669] Lustre: lustre-MDT0000: new disk, initializing [ 8477.932045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8477.974248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8482.230839] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8492.793056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8492.948715] Lustre: 443260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8493.002726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8493.007199] Lustre: Skipped 1 previous similar message [ 8493.066703] Lustre: lustre-MDT0001: new disk, initializing [ 8493.150417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8493.188188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8493.193914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8497.284908] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8506.587907] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8513.751772] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8522.264153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8522.588481] Lustre: lustre-OST0000: new disk, initializing [ 8522.591926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8522.600248] Lustre: 445229:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8522.689674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8527.970725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8527.977389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8528.136581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8529.613399] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8539.518195] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8548.837081] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8548.840186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8548.846624] Lustre: Skipped 2 previous similar messages [ 8548.854522] Lustre: Skipped 2 previous similar messages [ 8553.019402] Lustre: server umount lustre-MDT0000 complete [ 8556.763767] LustreError: 444063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786378290 with bad export cookie 518641349342657136 [ 8556.770553] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8556.783429] LustreError: 444063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8557.073778] Lustre: server umount lustre-MDT0001 complete [ 8570.940819] Lustre: server umount lustre-OST0000 complete [ 8581.236319] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:11:53 (1786378313) [ 8588.649788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8588.990520] LustreError: 447258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8589.008685] LustreError: 447258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8589.101631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8594.088346] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8594.402746] LustreError: 447259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8594.412404] LustreError: 447259:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8599.522412] LustreError: 447258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8602.400771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8602.641820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8607.546983] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8615.456351] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8621.197842] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8640.488416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8640.628611] Lustre: Found index 0 for lustre-OST0000, updating log [ 8640.774273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8647.210240] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 8654.890750] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8660.405122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8665.573856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8665.583847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8665.614219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8666.655652] Lustre: server umount lustre-OST0000 complete [ 8681.955766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8681.958783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8681.973256] Lustre: Skipped 2 previous similar messages [ 8681.990014] Lustre: Skipped 2 previous similar messages [ 8684.069298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8684.082363] Lustre: Skipped 3 previous similar messages [ 8689.136384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8689.141216] Lustre: Skipped 1 previous similar message [ 8696.288137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8696.465958] Lustre: server umount lustre-MDT0000 complete [ 8698.849157] LNet: 1 local NIs in recovery (showing 1): 192.168.203.148@tcp [ 8699.375131] LustreError: 447253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8699.390948] LustreError: 447253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8700.062963] LustreError: 447238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786378434 with bad export cookie 518641349342658529 [ 8700.072832] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8700.074476] LustreError: 447238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8700.660684] Lustre: server umount lustre-MDT0001 complete [ 8710.660807] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:14:02 (1786378442) [ 9317.124727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9317.427559] LustreError: 453669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9317.526462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9321.102738] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9323.024871] LustreError: 453670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9327.138139] LustreError: 453669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9327.463433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9327.707993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9331.147497] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9336.211941] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9340.269697] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9358.001933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9358.690957] Lustre: Found index 0 for lustre-OST0000, updating log [ 9359.381043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.763628] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9373.488907] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9376.001877] Lustre: server umount lustre-OST0000 complete [ 9376.417876] LNet: 1 local NIs in recovery (showing 1): 192.168.203.148@tcp [ 9381.739064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9381.750976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9381.765940] Lustre: Skipped 1 previous similar message [ 9381.793261] Lustre: Skipped 3 previous similar messages [ 9382.827029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9384.930444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9384.937583] Lustre: Skipped 4 previous similar messages [ 9386.954946] Lustre: server umount lustre-MDT0000 complete [ 9386.985226] LustreError: 453666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9386.995851] LustreError: 453666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9391.123719] LustreError: 453670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9391.182117] LustreError: 453670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9391.884391] LustreError: 455518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786379125 with bad export cookie 518641349342659614 [ 9391.886584] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9391.903019] LustreError: 455518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9392.129897] Lustre: server umount lustre-MDT0001 complete [ 9426.080534] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 9437.030193] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 9475.361566] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:26:47 (1786379207) [ 9494.099652] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 9497.281457] Key type lgssc unregistered [ 9497.474627] LNet: 460284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9497.485644] LNetError: 460284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9618.858710] LNet: Removed LNI 192.168.203.148@tcp [ 9619.613567] Key type .llcrypt unregistered [ 9619.620451] Key type ._llcrypt unregistered [ 9638.672447] Key type ._llcrypt registered [ 9638.674886] Key type .llcrypt registered [ 9638.788292] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 9640.730485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9640.755148] alg: No test for adler32 (adler32-zlib) [ 9641.713275] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 9641.889842] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 9643.514782] Key type lgssc registered [ 9644.326541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9648.862591] LNet: Added LNI 192.168.203.148@tcp100 [8/256/0/180] [ 9649.749068] LNet: Added LNI 192.168.203.148@tcp101 [8/256/0/180] [ 9650.582969] LNet: Added LNI 192.168.203.148@tcp102 [8/256/0/180] [ 9651.846175] LNet: Added LNI 192.168.203.148@tcp103 [8/256/0/180] [ 9652.629540] LNet: Added LNI 192.168.203.148@tcp104 [8/256/0/180] [ 9653.367956] LNet: Added LNI 192.168.203.148@tcp105 [8/256/0/180] [ 9654.254992] LNet: Added LNI 192.168.203.148@tcp106 [8/256/0/180] [ 9655.449042] LNet: Added LNI 192.168.203.148@tcp107 [8/256/0/180] [ 9656.604450] LNet: Added LNI 192.168.203.148@tcp108 [8/256/0/180] [ 9657.583567] LNet: Added LNI 192.168.203.148@tcp109 [8/256/0/180] [ 9658.335752] LNet: Added LNI 192.168.203.148@tcp110 [8/256/0/180] [ 9659.404698] LNet: Added LNI 192.168.203.148@tcp111 [8/256/0/180] [ 9660.312676] LNet: Added LNI 192.168.203.148@tcp112 [8/256/0/180] [ 9661.436391] LNet: Added LNI 192.168.203.148@tcp113 [8/256/0/180] [ 9662.790193] LNet: Added LNI 192.168.203.148@tcp114 [8/256/0/180] [ 9664.031758] LNet: Added LNI 192.168.203.148@tcp115 [8/256/0/180] [ 9665.772091] LNet: Added LNI 192.168.203.148@tcp116 [8/256/0/180] [ 9666.912502] LNet: Added LNI 192.168.203.148@tcp117 [8/256/0/180] [ 9667.794040] LNet: Added LNI 192.168.203.148@tcp118 [8/256/0/180] [ 9669.176985] LNet: Added LNI 192.168.203.148@tcp119 [8/256/0/180] [ 9670.244894] LNet: Added LNI 192.168.203.148@tcp120 [8/256/0/180] [ 9671.029420] LNet: Added LNI 192.168.203.148@tcp121 [8/256/0/180] [ 9671.801566] LNet: Added LNI 192.168.203.148@tcp122 [8/256/0/180] [ 9672.522898] LNet: Added LNI 192.168.203.148@tcp123 [8/256/0/180] [ 9673.650716] LNet: Added LNI 192.168.203.148@tcp124 [8/256/0/180] [ 9674.401331] LNet: Added LNI 192.168.203.148@tcp125 [8/256/0/180] [ 9675.136867] LNet: Added LNI 192.168.203.148@tcp126 [8/256/0/180] [ 9675.978046] LNet: Added LNI 192.168.203.148@tcp127 [8/256/0/180] [ 9677.058406] LNet: Added LNI 192.168.203.148@tcp128 [8/256/0/180] [ 9678.076490] LNet: Added LNI 192.168.203.148@tcp129 [8/256/0/180] [ 9679.352411] LNet: Added LNI 192.168.203.148@tcp130 [8/256/0/180] [ 9680.187426] LNet: Added LNI 192.168.203.148@tcp131 [8/256/0/180] [ 9680.906167] LNet: Added LNI 192.168.203.148@tcp132 [8/256/0/180] [ 9681.845513] LNet: Added LNI 192.168.203.148@tcp133 [8/256/0/180] [ 9682.588655] LNet: Added LNI 192.168.203.148@tcp134 [8/256/0/180] [ 9683.315315] LNet: Added LNI 192.168.203.148@tcp135 [8/256/0/180] [ 9690.611885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9690.650637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9691.737212] Lustre: 463323:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.148@tcp: can't add NID 192.168.203.148@tcp130: rc = -75 [ 9691.808143] LNetError: 463346:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.148 [ 9691.816596] LNetError: Protocol error connecting to 192.168.203.148@tcp117 at host 192.168.203.148:988: is it running a compatible version of Lustre? [ 9691.906436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9691.928630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9691.987994] Lustre: lustre-MDT0000: new disk, initializing [ 9692.055009] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9692.081609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9692.129023] LNet: 15 local NIs in recovery (showing 15): 192.168.203.148@tcp117, 192.168.203.148@tcp119, 192.168.203.148@tcp120, 192.168.203.148@tcp121, 192.168.203.148@tcp122, 192.168.203.148@tcp124, 192.168.203.148@tcp125, 192.168.203.148@tcp126, 192.168.203.148@tcp123, 192.168.203.148@tcp128, 192.168.203.148@tcp129, 192.168.203.148@tcp130, 192.168.203.148@tcp127, 192.168.203.148@tcp131, 192.168.203.148@tcp132 [ 9693.284161] LNetError: 460782:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.148 [ 9693.293889] LNetError: 460782:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 30 previous similar messages [ 9693.300497] LNetError: Protocol error connecting to 192.168.203.148@tcp131 at host 192.168.203.148:988: is it running a compatible version of Lustre? [ 9693.314044] LNetError: Skipped 30 previous similar messages [ 9694.307779] LNetError: 463353:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.148 [ 9694.317059] LNetError: 463353:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [ 9694.331919] LNetError: Protocol error connecting to 192.168.203.148@tcp117 at host 192.168.203.148:988: is it running a compatible version of Lustre? [ 9694.344306] LNetError: Skipped 4 previous similar messages [ 9696.885623] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9698.470105] LNetError: 463344:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.148 [ 9698.485307] LNetError: 463344:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 9698.501202] LNetError: Protocol error connecting to 192.168.203.148@tcp127 at host 192.168.203.148:988: is it running a compatible version of Lustre? [ 9698.530675] LNetError: Skipped 5 previous similar messages [ 9706.660764] LNetError: 463346:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.148 [ 9706.673119] LNetError: 463346:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 9706.682463] LNetError: Protocol error connecting to 192.168.203.148@tcp130 at host 192.168.203.148:988: is it running a compatible version of Lustre? [ 9706.698101] LNetError: Skipped 1 previous similar message [ 9708.729862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9708.823451] Lustre: 463340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9708.870570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9708.873930] Lustre: Skipped 1 previous similar message [ 9708.925384] Lustre: lustre-MDT0001: new disk, initializing [ 9708.957340] Lustre: 464141:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.148@tcp: can't add NID 192.168.203.148@tcp130: rc = -75 [ 9708.963552] Lustre: 464141:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9709.052312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9709.107598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9709.125831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9714.657201] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9715.943865] LNetError: 460782:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.148 [ 9715.948636] LNetError: 460782:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [ 9715.955578] LNetError: Protocol error connecting to 192.168.203.148@tcp133 at host 192.168.203.148:988: is it running a compatible version of Lustre? [ 9715.960969] LNetError: Skipped 10 previous similar messages [ 9716.111315] Lustre: 464470:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.148@tcp: can't add NID 192.168.203.148@tcp130: rc = -75 [ 9716.131429] Lustre: 464470:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9723.960531] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9729.974099] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9733.410148] LNetError: 463350:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.148 [ 9733.422060] LNetError: 463350:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 14 previous similar messages [ 9733.433095] LNetError: Protocol error connecting to 192.168.203.148@tcp128 at host 192.168.203.148:988: is it running a compatible version of Lustre? [ 9733.449675] LNetError: Skipped 14 previous similar messages [ 9738.391242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9738.878612] Lustre: lustre-OST0000: new disk, initializing [ 9738.882455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9738.888358] Lustre: 465316:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9738.962489] Lustre: 465321:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.148@tcp: can't add NID 192.168.203.148@tcp130: rc = -75 [ 9738.973931] Lustre: 465321:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 9739.051330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9745.043318] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9746.951706] Lustre: 465772:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.148@tcp: can't add NID 192.168.203.148@tcp130: rc = -75 [ 9746.963172] Lustre: 465772:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9747.049219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9747.078468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9747.130296] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9752.992142] LNet: 4 local NIs in recovery (showing 4): 192.168.203.148@tcp133, 192.168.203.148@tcp134, 192.168.203.148@tcp131, 192.168.203.148@tcp130 [ 9754.223719] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9765.417389] LNetError: 460780:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.148 [ 9765.426556] LNetError: 460780:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 29 previous similar messages [ 9767.433910] LNetError: Protocol error connecting to 192.168.203.148@tcp117 at host 192.168.203.148:988: is it running a compatible version of Lustre? [ 9767.456245] LNetError: Skipped 30 previous similar messages [ 9767.909858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9767.911085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9767.937407] LustreError: Skipped 1 previous similar message [ 9767.940623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9767.949879] Lustre: Skipped 1 previous similar message [ 9767.966421] Lustre: Skipped 1 previous similar message [ 9769.069385] Lustre: server umount lustre-OST0000 complete [ 9773.031782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9773.062238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9773.067550] Lustre: Skipped 1 previous similar message [ 9778.156177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9778.165971] Lustre: Skipped 1 previous similar message [ 9778.520978] Lustre: server umount lustre-MDT0000 complete [ 9781.927698] LustreError: 463332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786379515 with bad export cookie 4760575428099238387 [ 9781.934766] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9781.936146] LustreError: 463332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9782.201987] Lustre: server umount lustre-MDT0001 complete [ 9793.795681] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 9796.249623] Key type lgssc unregistered [ 9796.808323] LNet: 467230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9796.817939] LNetError: 467230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9796.823406] LNet: Removed LNI 192.168.203.148@tcp [ 9797.856218] LNet: Removed LNI 192.168.203.148@tcp100 [ 9797.861199] LNetError: 467230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9797.872242] LNetError: 467230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9797.882195] LNet: Removed LNI 192.168.203.148@tcp101 [ 9798.992019] LNet: Removed LNI 192.168.203.148@tcp102 [ 9798.996306] LNetError: 467230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9799.002276] LNetError: 467230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9799.007780] LNet: Removed LNI 192.168.203.148@tcp103 [ 9800.032146] LNet: Removed LNI 192.168.203.148@tcp104 [ 9801.056157] LNet: Removed LNI 192.168.203.148@tcp105 [ 9801.062971] LNetError: 467230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9801.074159] LNetError: 467230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9801.086888] LNet: Removed LNI 192.168.203.148@tcp106 [ 9802.144726] LNet: Removed LNI 192.168.203.148@tcp107 [ 9803.169540] LNet: Removed LNI 192.168.203.148@tcp108 [ 9804.194957] LNet: Removed LNI 192.168.203.148@tcp109 [ 9805.216151] LNet: Removed LNI 192.168.203.148@tcp110 [ 9805.218638] LNetError: 467230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9805.222758] LNetError: 467230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9805.227637] LNet: Removed LNI 192.168.203.148@tcp111 [ 9806.240192] LNet: Removed LNI 192.168.203.148@tcp112 [ 9807.264184] LNet: Removed LNI 192.168.203.148@tcp113 [ 9808.303633] LNet: Removed LNI 192.168.203.148@tcp114 [ 9809.312858] LNet: Removed LNI 192.168.203.148@tcp115 [ 9810.337283] LNet: Removed LNI 192.168.203.148@tcp116 [ 9811.425501] LNet: Removed LNI 192.168.203.148@tcp117 [ 9812.448799] LNet: Removed LNI 192.168.203.148@tcp118 [ 9813.472116] LNet: Removed LNI 192.168.203.148@tcp119 [ 9813.485474] LNetError: 467230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9813.490544] LNetError: 467230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 9814.496434] LNet: Removed LNI 192.168.203.148@tcp120 [ 9815.520140] LNet: Removed LNI 192.168.203.148@tcp121 [ 9816.544153] LNet: Removed LNI 192.168.203.148@tcp122 [ 9817.568153] LNet: Removed LNI 192.168.203.148@tcp123 [ 9818.592169] LNet: Removed LNI 192.168.203.148@tcp124 [ 9819.616152] LNet: Removed LNI 192.168.203.148@tcp125 [ 9820.640169] LNet: Removed LNI 192.168.203.148@tcp126 [ 9821.664329] LNet: Removed LNI 192.168.203.148@tcp127 [ 9822.688801] LNet: Removed LNI 192.168.203.148@tcp128 [ 9823.712161] LNet: Removed LNI 192.168.203.148@tcp129 [ 9823.714688] LNet: Removed LNI 192.168.203.148@tcp130 [ 9823.717496] LNet: Removed LNI 192.168.203.148@tcp131 [ 9824.736164] LNet: Removed LNI 192.168.203.148@tcp132 [ 9824.740668] LNet: Removed LNI 192.168.203.148@tcp133 [ 9824.746196] LNet: Removed LNI 192.168.203.148@tcp134 [ 9825.771295] LNet: Removed LNI 192.168.203.148@tcp135 [ 9826.361922] Key type .llcrypt unregistered [ 9826.371906] Key type ._llcrypt unregistered [ 9848.738575] Key type ._llcrypt registered [ 9848.744368] Key type .llcrypt registered [ 9848.832819] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [ 9861.398311] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 9862.461587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9862.497985] alg: No test for adler32 (adler32-zlib) [ 9863.475284] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 9863.756658] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 9865.552229] Key type lgssc registered [ 9866.902542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9903.470996] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:33:56 (1786379636) [ 9925.004958] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [ 9927.377744] Key type lgssc unregistered [ 9927.603861] LNet: 471300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9927.609741] LNetError: 471300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9927.621977] LNet: Removed LNI 192.168.203.148@tcp [ 9928.232566] Key type .llcrypt unregistered [ 9928.234788] Key type ._llcrypt unregistered [ 9946.896218] Key type ._llcrypt registered [ 9946.898665] Key type .llcrypt registered [ 9947.001717] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [ 9949.339847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9949.372093] alg: No test for adler32 (adler32-zlib) [ 9950.679356] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 9951.128468] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [ 9952.824230] Key type lgssc registered [ 9954.438539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9962.880078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9962.900320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9964.230166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9964.262464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9964.343094] Lustre: lustre-MDT0000: new disk, initializing [ 9964.424888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9964.439156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9969.963579] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9980.868467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9980.968489] Lustre: 472516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9981.008593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9981.011714] Lustre: Skipped 1 previous similar message [ 9981.062333] Lustre: lustre-MDT0001: new disk, initializing [ 9981.168420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9981.337861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9981.346947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9986.058235] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [ 9996.334800] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10002.788184] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10009.195327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10009.409777] Lustre: lustre-OST0000: new disk, initializing [10009.412936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10009.418033] Lustre: 474484:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10009.488050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10015.801650] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10016.318615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10016.344877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10016.564280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10024.755904] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10028.863623] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing lnet_if_list [10034.413777] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [10034.498267] LNet: Added LNI 192.168.203.148@tcp10 [8/256/0/180] [10040.693719] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [10040.716175] LNet: Added LNI 192.168.203.148@tcp11 [8/256/0/180] [10046.514577] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [10046.541352] LNet: Added LNI 192.168.203.148@tcp12 [8/256/0/180] [10050.766514] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [10050.804840] LNet: Added LNI 192.168.203.148@tcp13 [8/256/0/180] [10056.529771] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [10056.585268] LNet: Added LNI 192.168.203.148@tcp14 [8/256/0/180] [10062.414283] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [10062.435186] LNet: Added LNI 192.168.203.148@tcp15 [8/256/0/180] [10067.373717] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [10067.397509] LNet: Added LNI 192.168.203.148@tcp16 [8/256/0/180] [10072.916726] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10072.935419] LNet: Added LNI 192.168.203.148@tcp17 [8/256/0/180] [10077.734824] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [10077.762656] LNet: Added LNI 192.168.203.148@tcp18 [8/256/0/180] [10082.575447] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [10082.595461] LNet: Added LNI 192.168.203.148@tcp19 [8/256/0/180] [10090.457494] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [10090.487602] LNet: Added LNI 192.168.203.148@tcp20 [8/256/0/180] [10096.963335] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [10096.986837] LNet: Added LNI 192.168.203.148@tcp21 [8/256/0/180] [10101.892356] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [10101.926345] LNet: Added LNI 192.168.203.148@tcp22 [8/256/0/180] [10108.423573] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [10108.459970] LNet: Added LNI 192.168.203.148@tcp23 [8/256/0/180] [10115.888571] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [10115.911205] LNet: Added LNI 192.168.203.148@tcp24 [8/256/0/180] [10122.572370] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [10122.602246] LNet: Added LNI 192.168.203.148@tcp25 [8/256/0/180] [10128.814457] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [10128.834685] LNet: Added LNI 192.168.203.148@tcp26 [8/256/0/180] [10136.038936] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [10136.076731] LNet: Added LNI 192.168.203.148@tcp27 [8/256/0/180] [10142.213254] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [10142.234825] LNet: Added LNI 192.168.203.148@tcp28 [8/256/0/180] [10148.570764] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [10148.598973] LNet: Added LNI 192.168.203.148@tcp29 [8/256/0/180] [10155.388393] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [10155.408619] LNet: Added LNI 192.168.203.148@tcp30 [8/256/0/180] [10161.711788] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [10161.773446] LNet: Added LNI 192.168.203.148@tcp31 [8/256/0/180] [10170.234611] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [10170.286800] LNet: Added LNI 192.168.203.148@tcp32 [8/256/0/180] [10176.787555] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [10176.810362] LNet: Added LNI 192.168.203.148@tcp33 [8/256/0/180] [10184.323604] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10184.354124] LNet: Added LNI 192.168.203.148@tcp34 [8/256/0/180] [10191.842835] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10191.862964] LNet: Added LNI 192.168.203.148@tcp35 [8/256/0/180] [10199.840347] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10199.868987] LNet: Added LNI 192.168.203.148@tcp36 [8/256/0/180] [10208.891813] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10208.955475] LNet: Added LNI 192.168.203.148@tcp37 [8/256/0/180] [10216.363751] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10216.381128] LNet: Added LNI 192.168.203.148@tcp38 [8/256/0/180] [10224.420139] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10224.440428] LNet: Added LNI 192.168.203.148@tcp39 [8/256/0/180] [10232.573149] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10232.594939] LNet: Added LNI 192.168.203.148@tcp40 [8/256/0/180] [10240.024476] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10240.066971] LNet: Added LNI 192.168.203.148@tcp41 [8/256/0/180] [10247.764417] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10247.787523] LNet: Added LNI 192.168.203.148@tcp42 [8/256/0/180] [10256.610954] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10256.627315] LNet: Added LNI 192.168.203.148@tcp43 [8/256/0/180] [10267.810740] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10267.834576] LNet: Added LNI 192.168.203.148@tcp44 [8/256/0/180] [10276.430658] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10276.455391] LNet: Added LNI 192.168.203.148@tcp45 [8/256/0/180] [10289.540542] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10289.560736] LNet: Added LNI 192.168.203.148@tcp46 [8/256/0/180] [10301.059262] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10301.084472] LNet: Added LNI 192.168.203.148@tcp47 [8/256/0/180] [10313.471797] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10313.510716] LNet: Added LNI 192.168.203.148@tcp48 [8/256/0/180] [10324.111717] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10324.166698] LNet: Added LNI 192.168.203.148@tcp49 [8/256/0/180] [10334.802963] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10334.834468] LNet: Added LNI 192.168.203.148@tcp50 [8/256/0/180] [10355.695841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10355.713723] Lustre: Skipped 2 previous similar messages [10355.713987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10355.716401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10355.732975] Lustre: Skipped 2 previous similar messages [10360.503876] Lustre: server umount lustre-MDT0000 complete [10364.414677] LustreError: 474483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786380098 with bad export cookie 1519380251011410229 [10364.417477] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10364.429426] LustreError: 474483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10364.722475] Lustre: server umount lustre-MDT0001 complete [10381.313929] Lustre: 471788:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786380099/real 1786380099] req@00000000e3ba4a38 x1873154864757632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786380115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10381.332298] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10385.380883] Lustre: 471789:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786380103/real 1786380103] req@0000000064f5b5da x1873154864757760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786380119 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10386.324426] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [10386.401988] Lustre: 471786:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786380104/real 1786380104] req@00000000a9c726b0 x1873154864757888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786380120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10390.560085] Lustre: 471787:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786380108/real 1786380108] req@0000000016fa4452 x1873154864758144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786380124 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10395.680780] Lustre: 471789:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786380113/real 1786380113] req@000000002c8f7962 x1873154864758272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786380129 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10396.646866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10409.057144] LustreError: 471785:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001b5d089a x1873154864759040/t0(0) o250->MGC192.168.203.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10409.296558] LustreError: 474477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10409.320247] LustreError: 474477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10409.397040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10410.417205] LustreError: 474478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10410.456203] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10412.462253] LustreError: 474477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10414.551354] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10417.653375] LustreError: 474478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10422.772836] LustreError: 474477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.654898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.153813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10426.224683] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10430.378990] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10438.592199] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10444.431733] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10466.283917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10466.284266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10466.287774] LustreError: Skipped 1 previous similar message [10466.289142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10466.294612] Lustre: Skipped 1 previous similar message [10466.304109] Lustre: Skipped 1 previous similar message [10467.329847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10467.332973] Lustre: Skipped 1 previous similar message [10470.790688] Lustre: server umount lustre-OST0000 complete [10476.517316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10476.545483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10476.560159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10480.922272] Lustre: server umount lustre-MDT0000 complete [10482.668257] LustreError: 498732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10482.683272] LustreError: 498732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10484.925591] LustreError: 474483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786380218 with bad export cookie 1519380251011412812 [10484.945869] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10484.946373] LustreError: 474483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10485.320701] Lustre: server umount lustre-MDT0001 complete [10495.401962] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [10498.401732] Key type lgssc unregistered [10498.825255] LNet: 501536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10498.832565] LNetError: 501536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10498.864372] LNet: Removed LNI 192.168.203.148@tcp [10498.867775] LNet: Removed LNI 192.168.203.148@tcp10 [10498.871396] LNet: Removed LNI 192.168.203.148@tcp11 [10498.875325] LNet: Removed LNI 192.168.203.148@tcp12 [10498.878565] LNet: Removed LNI 192.168.203.148@tcp13 [10498.882265] LNet: Removed LNI 192.168.203.148@tcp14 [10498.886416] LNet: Removed LNI 192.168.203.148@tcp15 [10498.890553] LNet: Removed LNI 192.168.203.148@tcp16 [10498.893436] LNet: Removed LNI 192.168.203.148@tcp17 [10498.896613] LNet: Removed LNI 192.168.203.148@tcp18 [10498.899425] LNet: Removed LNI 192.168.203.148@tcp19 [10498.902254] LNet: Removed LNI 192.168.203.148@tcp20 [10498.905850] LNet: Removed LNI 192.168.203.148@tcp21 [10498.909563] LNet: Removed LNI 192.168.203.148@tcp22 [10498.912447] LNet: Removed LNI 192.168.203.148@tcp23 [10498.916650] LNet: Removed LNI 192.168.203.148@tcp24 [10498.939377] LNet: Removed LNI 192.168.203.148@tcp25 [10498.948628] LNet: Removed LNI 192.168.203.148@tcp26 [10498.955151] LNet: Removed LNI 192.168.203.148@tcp27 [10498.957639] LNet: Removed LNI 192.168.203.148@tcp28 [10498.962311] LNet: Removed LNI 192.168.203.148@tcp29 [10498.965430] LNet: Removed LNI 192.168.203.148@tcp30 [10498.969351] LNet: Removed LNI 192.168.203.148@tcp31 [10498.972410] LNet: Removed LNI 192.168.203.148@tcp32 [10498.976347] LNet: Removed LNI 192.168.203.148@tcp33 [10498.979797] LNet: Removed LNI 192.168.203.148@tcp34 [10498.982518] LNet: Removed LNI 192.168.203.148@tcp35 [10498.985283] LNet: Removed LNI 192.168.203.148@tcp36 [10498.987891] LNet: Removed LNI 192.168.203.148@tcp37 [10498.990550] LNet: Removed LNI 192.168.203.148@tcp38 [10498.993602] LNet: Removed LNI 192.168.203.148@tcp39 [10498.996170] LNet: Removed LNI 192.168.203.148@tcp40 [10498.997743] LNet: Removed LNI 192.168.203.148@tcp41 [10498.999800] LNet: Removed LNI 192.168.203.148@tcp42 [10499.002767] LNet: Removed LNI 192.168.203.148@tcp43 [10499.005545] LNet: Removed LNI 192.168.203.148@tcp44 [10499.009289] LNet: Removed LNI 192.168.203.148@tcp45 [10499.015378] LNet: Removed LNI 192.168.203.148@tcp46 [10499.019611] LNet: Removed LNI 192.168.203.148@tcp47 [10499.024880] LNet: Removed LNI 192.168.203.148@tcp48 [10499.029630] LNet: Removed LNI 192.168.203.148@tcp49 [10499.039217] LNet: Removed LNI 192.168.203.148@tcp50 [10500.002341] Key type .llcrypt unregistered [10500.007784] Key type ._llcrypt unregistered [10524.760806] Key type ._llcrypt registered [10524.763374] Key type .llcrypt registered [10524.851467] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [10539.990682] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [10541.529407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10541.575017] alg: No test for adler32 (adler32-zlib) [10542.758054] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [10542.944439] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [10544.648211] Key type lgssc registered [10545.605384] Lustre: Echo OBD driver; http://www.lustre.org/ [10579.349909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10579.376675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10580.549729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10580.575574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10580.628324] Lustre: lustre-MDT0000: new disk, initializing [10580.685414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10580.698931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10585.464449] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10595.851591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10595.917669] Lustre: 504734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10595.972624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10595.975420] Lustre: Skipped 1 previous similar message [10596.171788] Lustre: lustre-MDT0001: new disk, initializing [10596.455378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10596.492526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10596.503233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10601.172289] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10612.344665] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10617.599729] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10626.801319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10626.963591] Lustre: *** cfs_fail_loc=90f, val=0*** [10627.021918] Lustre: lustre-OST0000: new disk, initializing [10627.026748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10627.034977] Lustre: 506752:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10627.111909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10633.529573] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10634.981648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10635.008718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10635.340485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10644.024946] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10655.714878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10655.722187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10655.725467] LustreError: Skipped 1 previous similar message [10655.729417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10655.769969] Lustre: Skipped 1 previous similar message [10658.524334] Lustre: server umount lustre-OST0000 complete [10662.884875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10662.897961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10662.927342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10662.939530] Lustre: Skipped 1 previous similar message [10665.954253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10665.958589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10668.995138] Lustre: server umount lustre-MDT0000 complete [10672.394189] LustreError: 504726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786380406 with bad export cookie 3201515649953334969 [10672.402409] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10672.421480] LustreError: 504726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10672.769153] Lustre: server umount lustre-MDT0001 complete [10685.835496] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [10688.429760] Key type lgssc unregistered [10688.645858] LNet: 508663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10688.651517] LNetError: 508663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10689.712825] LNet: Removed LNI 192.168.203.148@tcp [10690.461291] Key type .llcrypt unregistered [10690.463256] Key type ._llcrypt unregistered [10710.546482] Key type ._llcrypt registered [10710.549168] Key type .llcrypt registered [10710.657000] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [10711.770018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10711.852142] alg: No test for adler32 (adler32-zlib) [10713.035162] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [10713.365946] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [10715.008175] Key type lgssc registered [10716.152684] Lustre: Echo OBD driver; http://www.lustre.org/ [10724.391953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10724.407544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10725.708651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10729.658289] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10735.078631] LustreError: 509935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10735.110026] LustreError: 509935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10738.327350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10738.575620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10742.981589] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10750.430950] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10756.658543] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10762.844590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10763.095723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10768.980872] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10776.938711] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10787.129817] loop0: detected capacity change from 0 to 204800000 [10787.350491] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10787.363976] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10787.407343] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10794.436205] loop0: detected capacity change from 0 to 204800000 [10794.584881] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10794.597230] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10794.610304] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10800.340091] loop0: detected capacity change from 0 to 204800000 [10806.259149] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10806.423627] Lustre: fs15246-OST0000: new disk, initializing [10806.426449] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10806.433525] Lustre: 513523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10806.536259] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10813.201764] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10822.638588] loop1: detected capacity change from 0 to 204800000 [10828.440465] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10828.643171] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10828.702311] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10828.852935] Lustre: fs15246-MDT0000: new disk, initializing [10829.006569] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10829.026284] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10829.133311] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10830.178135] Lustre: 514646:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000001b2139d3 x1873155663641984/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786380580 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10833.417886] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [10839.289652] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10839.316626] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10839.340495] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10859.495717] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10859.518286] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10864.087428] Lustre: server umount fs15246-MDT0000 complete [10864.618324] LustreError: 511373:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10869.782269] LustreError: 511373:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.229813] Lustre: server umount fs15246-OST0000 complete [10885.096447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10885.108837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10885.125380] Lustre: Skipped 1 previous similar message [10885.139702] Lustre: Skipped 1 previous similar message [10890.212280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10890.226104] Lustre: Skipped 1 previous similar message [10891.163970] Lustre: server umount lustre-OST0000 complete [10895.330673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10895.334704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10895.354653] Lustre: Skipped 1 previous similar message [10895.368332] Lustre: Skipped 1 previous similar message [10900.393925] Lustre: server umount lustre-MDT0000 complete [10900.451714] LustreError: 509935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10900.468429] LustreError: 509935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10904.066937] LustreError: 509914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786380638 with bad export cookie 717851202582672749 [10904.092427] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10904.637264] Lustre: server umount lustre-MDT0001 complete [10917.429202] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [10920.080638] Key type lgssc unregistered [10920.289204] LNet: 517748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10920.308899] LNetError: 517748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10920.324862] LNet: Removed LNI 192.168.203.148@tcp [10920.896278] Key type .llcrypt unregistered [10920.901709] Key type ._llcrypt unregistered [10937.869728] Key type ._llcrypt registered [10937.872457] Key type .llcrypt registered [10937.953594] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [10939.245141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10939.352643] alg: No test for adler32 (adler32-zlib) [10940.410569] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [10940.644792] LNet: Added LNI 192.168.203.148@tcp [8/256/0/180] [10942.288198] Key type lgssc registered [10943.077985] Lustre: Echo OBD driver; http://www.lustre.org/ [10978.713379] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [10988.073975] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [11028.326611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11028.353455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11029.715828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11029.756971] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11029.807824] Lustre: lustre-MDT0000: new disk, initializing [11029.859353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11029.881225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11034.080468] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11047.520353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11047.622452] Lustre: 522200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11047.675485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11047.682369] Lustre: Skipped 1 previous similar message [11047.758989] Lustre: lustre-MDT0001: new disk, initializing [11047.877248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11048.015549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11048.037795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11053.651653] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11063.220552] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11069.054912] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11077.853650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11078.239106] Lustre: lustre-OST0000: new disk, initializing [11078.243074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11078.248798] Lustre: 524171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11078.341521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11084.891584] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_default_debug -1 all [11085.849042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11085.859651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11086.029211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11094.603397] Lustre: DEBUG MARKER: oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11099.093862] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11099.248152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11103.543094] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11103.775949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11106.276334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11106.290156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11106.303903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11106.790809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11106.793167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11106.825303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11111.907351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11111.911978] Lustre: Skipped 1 previous similar message [11117.044336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11117.091557] Lustre: Skipped 1 previous similar message [11120.096147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11120.198505] Lustre: server umount lustre-OST0000 complete [11124.705623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11124.714430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11124.739406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11127.265857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11130.564093] Lustre: server umount lustre-MDT0000 complete [11133.965544] LustreError: 522192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786380867 with bad export cookie 9307108723438400582 [11133.971972] LustreError: MGC192.168.203.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11133.983122] LustreError: 522192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11134.010605] LustreError: 518261:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [11134.347905] Lustre: server umount lustre-MDT0001 complete [11162.548897] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing set_hostid [11171.284395] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing load_modules_local [11201.884603] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10954 sec ========= 12:55:34 (1786380934) [11204.356916] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:55:36 (1786380936) === [11207.822947] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:55:40 (1786380940) === [11250.478345] Lustre: DEBUG MARKER: oleg348-server.virtnet: executing unload_modules_local [11253.342803] Key type lgssc unregistered [11253.561707] LNet: 530816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11253.576912] LNetError: 530816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11253.589984] LNet: Removed LNI 192.168.203.148@tcp [11254.363616] Key type .llcrypt unregistered [11254.367685] Key type ._llcrypt unregistered