[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 472775057 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003269] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008693] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011142] LSM: Security Framework initializing [ 0.012065] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.015078] *** VALIDATE selinux *** [ 0.023536] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028298] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030129] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031126] *** VALIDATE tmpfs *** [ 0.033128] *** VALIDATE proc *** [ 0.034277] *** VALIDATE cgroup *** [ 0.035013] *** VALIDATE cgroup2 *** [ 0.036299] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039024] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.042718] debug: unmapping init [mem 0xffffffff8ca59000-0xffffffff8ca60fff] [ 0.044205] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045725] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046028] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.054200] rcu: Hierarchical SRCU implementation. [ 0.056362] smp: Bringing up secondary CPUs ... [ 0.057625] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.061551] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064021] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.258479] node 0 deferred pages initialised in 192ms [ 0.260581] devtmpfs: initialized [ 0.261348] x86/mm: Memory block size: 128MB [ 0.264634] gcov: version magic: 0x41383552 [ 0.268052] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.269118] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.271510] pinctrl core: initialized pinctrl subsystem [ 0.273259] [ 0.273820] ************************************************************* [ 0.276012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.278015] ** ** [ 0.280016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.282015] ** ** [ 0.285015] ** This means that this kernel is built to expose internal ** [ 0.287014] ** IOMMU data structures, which may compromise security on ** [ 0.289015] ** your system. ** [ 0.291018] ** ** [ 0.293015] ** If you see this message and you are not debugging the ** [ 0.295017] ** kernel, report this immediately to your vendor! ** [ 0.297016] ** ** [ 0.299028] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.302021] ************************************************************* [ 0.304517] NET: Registered protocol family 16 [ 0.306519] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.309086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.312093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.316051] cpuidle: using governor menu [ 0.318000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.320627] PCI: Using configuration type 1 for base access [ 0.323158] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.332149] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.334039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.337208] cryptd: max_cpu_qlen set to 1000 [ 0.339364] ACPI: Added _OSI(Module Device) [ 0.341025] ACPI: Added _OSI(Processor Device) [ 0.342019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.344021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.349162] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.356179] ACPI: Interpreter enabled [ 0.357082] ACPI: PM: (supports S0 S3 S4 S5) [ 0.359025] ACPI: Using IOAPIC for interrupt routing [ 0.361140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.364463] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.375873] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.378067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.380024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.384109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.389441] acpiphp: Slot [2] registered [ 0.391148] acpiphp: Slot [5] registered [ 0.392152] acpiphp: Slot [6] registered [ 0.393159] acpiphp: Slot [7] registered [ 0.395204] acpiphp: Slot [8] registered [ 0.397172] acpiphp: Slot [9] registered [ 0.398136] acpiphp: Slot [10] registered [ 0.399111] acpiphp: Slot [3] registered [ 0.400037] acpiphp: Slot [4] registered [ 0.401073] acpiphp: Slot [11] registered [ 0.403157] acpiphp: Slot [12] registered [ 0.404099] acpiphp: Slot [13] registered [ 0.406159] acpiphp: Slot [14] registered [ 0.407119] acpiphp: Slot [15] registered [ 0.408157] acpiphp: Slot [16] registered [ 0.410117] acpiphp: Slot [17] registered [ 0.411104] acpiphp: Slot [18] registered [ 0.412134] acpiphp: Slot [19] registered [ 0.414124] acpiphp: Slot [20] registered [ 0.415176] acpiphp: Slot [21] registered [ 0.417135] acpiphp: Slot [22] registered [ 0.418128] acpiphp: Slot [23] registered [ 0.420133] acpiphp: Slot [24] registered [ 0.421096] acpiphp: Slot [25] registered [ 0.422116] acpiphp: Slot [26] registered [ 0.424134] acpiphp: Slot [27] registered [ 0.425134] acpiphp: Slot [28] registered [ 0.426129] acpiphp: Slot [29] registered [ 0.428139] acpiphp: Slot [30] registered [ 0.429158] acpiphp: Slot [31] registered [ 0.431072] PCI host bridge to bus 0000:00 [ 0.432024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.435036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.438033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.441046] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.444035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.447040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.449224] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.453084] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.456396] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.466023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.471095] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.474026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.475030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.477026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.481187] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.482784] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.485048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.488818] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.493024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.506022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.510019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.515400] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.521021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.529028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.543024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.552304] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.558022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.565026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.584027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.593856] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.601022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.606021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.621034] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.630444] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.635019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.640023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.657026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.666166] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.672025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.677024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.692024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.701507] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.707019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.713025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.728032] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.740086] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.742419] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.744456] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.747548] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.750364] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.756123] iommu: Default domain type: Passthrough [ 0.758606] SCSI subsystem initialized [ 0.759190] ACPI: bus type USB registered [ 0.761141] usbcore: registered new interface driver usbfs [ 0.763134] usbcore: registered new interface driver hub [ 0.765114] usbcore: registered new device driver usb [ 0.766214] pps_core: LinuxPPS API ver. 1 registered [ 0.768017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.771085] PTP clock support registered [ 0.774082] EDAC MC: Ver: 3.0.0 [ 0.775434] PCI: Using ACPI for IRQ routing [ 0.776792] NetLabel: Initializing [ 0.778023] NetLabel: domain hash size = 128 [ 0.779012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.781095] NetLabel: unlabeled traffic allowed by default [ 0.782141] vgaarb: loaded [ 0.783336] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.785015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.791170] clocksource: Switched to clocksource kvm-clock [ 0.881758] VFS: Disk quotas dquot_6.6.0 [ 0.883114] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.885049] *** VALIDATE ramfs *** [ 0.886175] *** VALIDATE hugetlbfs *** [ 0.887524] pnp: PnP ACPI init [ 0.890226] pnp: PnP ACPI: found 6 devices [ 0.907678] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.910447] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.912841] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.915140] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.916873] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.918895] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.920808] NET: Registered protocol family 2 [ 0.922836] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.926524] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.929454] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.935110] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.938940] TCP: Hash tables configured (established 65536 bind 65536) [ 0.942284] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.945870] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.949273] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.952407] NET: Registered protocol family 1 [ 0.955066] RPC: Registered named UNIX socket transport module. [ 0.957346] RPC: Registered udp transport module. [ 0.958818] RPC: Registered tcp transport module. [ 0.960787] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.963478] NET: Registered protocol family 44 [ 0.964624] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.966433] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.968456] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.970056] PCI: CLS 0 bytes, default 64 [ 0.971245] Unpacking initramfs... [ 2.390695] debug: unmapping init [mem 0xffff97e33cc54000-0xffff97e33ffbffff] [ 2.394389] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.396701] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.399491] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 2.920412] Initialise system trusted keyrings [ 2.922263] Key type blacklist registered [ 2.924499] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.933636] zbud: loaded [ 2.937069] *** VALIDATE nfs *** [ 2.938330] *** VALIDATE nfs4 *** [ 2.939775] pstore: using deflate compression [ 2.942876] Platform Keyring initialized [ 3.064706] NET: Registered protocol family 38 [ 3.066218] Key type asymmetric registered [ 3.067802] Asymmetric key parser 'x509' registered [ 3.069798] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.073132] io scheduler mq-deadline registered [ 3.074951] io scheduler kyber registered [ 3.076866] io scheduler bfq registered [ 3.078882] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.082304] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.085528] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.088765] ACPI: Power Button [PWRF] [ 3.096329] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.103400] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.119627] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.127360] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.143363] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.173492] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.203805] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.209674] Non-volatile memory driver v1.3 [ 3.211456] Linux agpgart interface v0.103 [ 3.242419] virtio_blk virtio1: [vda] 135992 512-byte logical blocks (69.6 MB/66.4 MiB) [ 3.244281] vda: detected capacity change from 0 to 69627904 [ 3.257153] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.260255] vdb: detected capacity change from 0 to 1073741824 [ 3.275150] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.278317] vdc: detected capacity change from 0 to 2621440000 [ 3.292905] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.295960] vdd: detected capacity change from 0 to 2621440000 [ 3.311405] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.314461] vde: detected capacity change from 0 to 4294967296 [ 3.332972] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.336447] vdf: detected capacity change from 0 to 4294967296 [ 3.344525] libphy: Fixed MDIO Bus: probed [ 3.350762] usbcore: registered new interface driver usbserial_generic [ 3.353545] usbserial: USB Serial support registered for generic [ 3.356269] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.360672] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.362742] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.365487] mousedev: PS/2 mouse device common for all mice [ 3.368805] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.370513] rtc_cmos 00:05: RTC can wake from S4 [ 3.375282] rtc_cmos 00:05: registered as rtc0 [ 3.377681] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.378309] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.381211] intel_pstate: CPU model not supported [ 3.388052] hid: raw HID events driver (C) Jiri Kosina [ 3.389151] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.390601] usbcore: registered new interface driver usbhid [ 3.395746] usbhid: USB HID core driver [ 3.397465] drop_monitor: Initializing network drop monitor service [ 3.400024] Initializing XFRM netlink socket [ 3.402217] NET: Registered protocol family 10 [ 3.405398] Segment Routing with IPv6 [ 3.407099] NET: Registered protocol family 17 [ 3.409466] mpls_gso: MPLS GSO support [ 3.415368] RAS: Correctable Errors collector initialized. [ 3.417920] AVX version of gcm_enc/dec engaged. [ 3.420277] AES CTR mode by8 optimization enabled [ 3.498342] sched_clock: Marking stable (3498313833, 0)->(4409373115, -911059282) [ 3.502446] registered taskstats version 1 [ 3.504612] Loading compiled-in X.509 certificates [ 3.506629] zswap: loaded using pool lzo/zbud [ 3.530797] Key type big_key registered [ 3.545078] Key type encrypted registered [ 3.546600] ima: No TPM chip found, activating TPM-bypass! [ 3.548481] ima: Allocated hash algorithm: sha1 [ 3.550380] ima: No architecture policies found [ 3.552294] evm: Initialising EVM extended attributes: [ 3.554180] evm: security.selinux [ 3.555364] evm: security.ima [ 3.556441] evm: security.capability [ 3.557770] evm: HMAC attrs: 0x1 [ 3.560317] rtc_cmos 00:05: setting system clock to 2026-06-07 17:01:53 UTC (1780851713) [ 3.569465] debug: unmapping init [mem 0xffffffff8da03000-0xffffffff8dbfffff] [ 3.572528] debug: unmapping init [mem 0xffffffff8c782000-0xffffffff8ca58fff] [ 3.581105] Write protecting the kernel read-only data: 28672k [ 3.584574] debug: unmapping init [mem 0xffffffff8ae03000-0xffffffff8affffff] [ 3.587532] debug: unmapping init [mem 0xffffffff8b714000-0xffffffff8b7fffff] [ 3.621848] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.628989] systemd[1]: Detected virtualization kvm. [ 3.630920] systemd[1]: Detected architecture x86-64. [ 3.632301] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.658466] systemd[1]: No hostname configured. [ 3.659661] systemd[1]: Set hostname to . [ 3.661142] random: systemd: uninitialized urandom read (16 bytes read) [ 3.662861] systemd[1]: Initializing machine ID from random generator. [ 3.695177] random: ln: uninitialized urandom read (6 bytes read) [ 3.803742] random: systemd: uninitialized urandom read (16 bytes read) [ 3.806871] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.812858] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.817341] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.451749] device-mapper: uevent: version 1.0.3 [ 4.454111] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.203392] random: fast init done [ 5.242380] virtio_net virtio0 ens2: renamed from eth0 [ 5.253165] scsi host0: ata_piix [ 5.310353] scsi host1: ata_piix [ 5.312119] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.314856] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.093304] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.064182] random: crng init done [ 10.065546] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.523126] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.712571] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.007533] SELinux: Disabled at runtime. [ 12.075088] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.083346] systemd[1]: Detected virtualization kvm. [ 12.085194] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.650223] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.655297] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.661312] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.666349] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.670696] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.679540] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.683400] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ 12.759739] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.173210] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.507109] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.527211] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.733645] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.744502] EDAC sbridge: Ver: 1.1.2 [ 15.671139] Key type dns_resolver registered [ 15.986388] NFS: Registering the id_resolver key type [ 15.988391] Key type id_resolver registered [ 15.990022] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg204-server login: [ 71.475191] hrtimer: interrupt took 3369937 ns [ 100.186617] libcfs: loading out-of-tree module taints kernel. [ 100.333496] Key type ._llcrypt registered [ 100.339720] Key type .llcrypt registered [ 100.549391] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 122.451050] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 124.063984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 124.091076] alg: No test for adler32 (adler32-zlib) [ 125.715160] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 126.537348] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 128.490122] Key type lgssc registered [ 131.145606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 154.102886] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 210.506331] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 229.380876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 229.430361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.966725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 231.043161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 231.241682] Lustre: lustre-MDT0000: new disk, initializing [ 231.455994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.524260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 238.056744] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.485334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 254.596549] Lustre: 6512:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 254.624343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 254.627927] Lustre: Skipped 1 previous similar message [ 254.706189] Lustre: lustre-MDT0001: new disk, initializing [ 254.790485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 254.836339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 254.858556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 259.497477] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.001905] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 276.137474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 276.476839] Lustre: lustre-OST0000: new disk, initializing [ 276.479940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 276.484159] Lustre: 8418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 276.585700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 279.681213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 279.694924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 279.797596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 283.924629] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 299.056590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 299.162386] Lustre: lustre-OST0001: new disk, initializing [ 299.164994] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 299.169911] Lustre: 9473:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 299.225154] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 305.903902] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 308.803904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 308.815349] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 308.862435] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 319.845441] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 329.444943] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 335.722968] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing check_logdir /tmp/testlogs/ [ 343.152303] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing yml_node [ 349.468972] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 352.518909] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 356.554592] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 359.072339] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Jun 7 13:07:48 EDT 2026 [ 380.123890] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 382.611132] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 401.378292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.383399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 401.414505] Lustre: Skipped 3 previous similar messages [ 401.449426] Lustre: Skipped 3 previous similar messages [ 404.925419] Lustre: server umount lustre-MDT0000 complete [ 410.128943] LustreError: 10271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780852120 with bad export cookie 17864279217944301098 [ 410.129749] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 410.139636] LustreError: 10271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 411.616679] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.617914] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 411.642489] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 411.672127] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 411.690533] Lustre: Skipped 1 previous similar message [ 415.649948] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.650381] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 415.681313] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 415.705266] Lustre: Skipped 1 previous similar message [ 416.547238] Lustre: server umount lustre-MDT0001 complete [ 422.279433] Lustre: server umount lustre-OST0000 complete [ 426.974351] Lustre: server umount lustre-OST0001 complete [ 434.743860] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 446.912100] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 490.422028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.719475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 490.750348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 490.824397] Lustre: lustre-MDT0000: new disk, initializing [ 490.905339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.926582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 495.938520] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 509.094697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.219346] Lustre: 14993:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 509.278898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 509.287363] Lustre: Skipped 1 previous similar message [ 509.425553] Lustre: lustre-MDT0001: new disk, initializing [ 509.539842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 509.583453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 509.599084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 515.412378] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 524.416874] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 528.872669] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 536.286348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 536.660322] Lustre: lustre-OST0000: new disk, initializing [ 536.664225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 536.678816] Lustre: 16738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 536.787296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.796442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 537.816200] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 537.977563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 542.834357] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 551.539989] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 557.695594] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 557.946500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 562.907605] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 563.182995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 568.805807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 568.807892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.822797] LustreError: Skipped 1 previous similar message [ 568.848054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 568.858291] Lustre: Skipped 2 previous similar messages [ 568.890923] Lustre: Skipped 1 previous similar message [ 571.606636] Lustre: server umount lustre-OST0000 complete [ 579.063558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.077130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 579.083636] Lustre: Skipped 1 previous similar message [ 582.754666] Lustre: server umount lustre-MDT0000 complete [ 584.163122] LustreError: 15006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.182427] LustreError: 15006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 587.463405] LustreError: 16737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780852297 with bad export cookie 17864279217944302631 [ 587.476720] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.488694] LustreError: 16737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 587.990773] Lustre: server umount lustre-MDT0001 complete [ 593.403439] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:11:42 (1780852302) [ 596.523741] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 599.295342] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:11:48 (1780852308) [ 606.604803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.170221] LustreError: 19001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.289744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 612.323144] LustreError: 19002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.397612] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 622.499779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.569863] LustreError: 19001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 622.607246] LustreError: 19001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 627.645744] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 633.383392] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 637.515837] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 644.108725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 650.902958] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 657.378674] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 668.132368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 668.136276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 668.153473] LustreError: Skipped 1 previous similar message [ 668.158087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 668.187797] Lustre: Skipped 2 previous similar messages [ 672.391610] Lustre: server umount lustre-OST0000 complete [ 678.371151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 678.397881] Lustre: Skipped 1 previous similar message [ 683.500890] LustreError: 19977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 683.544432] LustreError: 19977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 683.766740] Lustre: server umount lustre-MDT0000 complete [ 688.895807] LustreError: 18982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780852398 with bad export cookie 17864279217944303513 [ 688.901202] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 688.906504] LustreError: 18982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 689.404889] Lustre: server umount lustre-MDT0001 complete [ 706.852518] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 710.230399] Key type lgssc unregistered [ 710.480878] LNet: 22195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.488930] LNetError: 22195:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.527669] LNet: Removed LNI 192.168.202.104@tcp [ 712.488369] Key type .llcrypt unregistered [ 712.490153] Key type ._llcrypt unregistered [ 744.820715] Key type ._llcrypt registered [ 744.823918] Key type .llcrypt registered [ 744.931652] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 746.585989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 746.780981] alg: No test for adler32 (adler32-zlib) [ 747.979709] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 748.322190] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 750.008180] Key type lgssc registered [ 751.489919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 760.394705] loop: module loaded [ 760.510153] loop0: detected capacity change from 0 to 204800000 [ 760.827725] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 760.839018] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 761.391892] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 769.136346] loop0: detected capacity change from 0 to 204800000 [ 769.371486] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 769.407600] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 769.422215] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 776.003743] loop0: detected capacity change from 0 to 204800000 [ 783.171977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 783.194286] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.523755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 784.572607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 784.683343] Lustre: lustre-MDT0000: new disk, initializing [ 784.791615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 784.831890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 789.666702] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 800.183431] loop1: detected capacity change from 0 to 204800000 [ 807.629802] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 808.151600] Lustre: lustre-OST07c6: new disk, initializing [ 808.159649] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 808.172555] Lustre: Skipped 1 previous similar message [ 808.182258] Lustre: 25351:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 808.333601] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 816.514875] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 816.781423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 816.794200] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 816.844360] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 821.709270] Lustre: 24354:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 821.734525] Lustre: 24354:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 821.746382] Lustre: 24354:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 821.772143] Lustre: 24354:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 821.783598] Lustre: 24354:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 821.799435] Lustre: 24354:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 837.757449] Lustre: 24355:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 837.763340] Lustre: 24355:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 837.768073] Lustre: 24355:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 837.776886] Lustre: 24355:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 837.783333] Lustre: 24355:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 837.788487] Lustre: 24355:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 843.040863] Lustre: 24355:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 843.053077] Lustre: 24355:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 843.057740] Lustre: 24355:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 843.069751] Lustre: 24355:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 843.076815] Lustre: 24355:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 843.084391] Lustre: 24355:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 843.089979] Lustre: 24355:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 843.097238] Lustre: 24355:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 843.103927] Lustre: 24355:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 843.110426] Lustre: 24355:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 843.117880] Lustre: 24355:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 843.125274] Lustre: 24355:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 846.322816] Lustre: 24353:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 846.345416] Lustre: 24353:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 846.361905] Lustre: 24353:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 846.369576] Lustre: 24353:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 846.387309] Lustre: 24353:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 846.402300] Lustre: 24353:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 846.417258] Lustre: 24353:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 846.429754] Lustre: 24353:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 846.442839] Lustre: 24353:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 846.457552] Lustre: 24353:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 846.469061] Lustre: 24353:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 846.479771] Lustre: 24353:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.666483] Lustre: 24353:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 851.693757] Lustre: 24353:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 851.708657] Lustre: 24353:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 851.728904] Lustre: 24353:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.742449] Lustre: 24353:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 851.757929] Lustre: 24353:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.777876] Lustre: 24353:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 851.796951] Lustre: 24353:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.809204] Lustre: 24353:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 851.820523] Lustre: 24353:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 851.836569] Lustre: 24353:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 851.844984] Lustre: 24353:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 860.299443] Lustre: 24354:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 860.314672] Lustre: 24354:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 860.326230] Lustre: 24354:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 860.338392] Lustre: 24354:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 860.349880] Lustre: 24354:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 860.361264] Lustre: 24354:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 860.371981] Lustre: 24354:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 860.379904] Lustre: 24354:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 860.390391] Lustre: 24354:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 860.399665] Lustre: 24354:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 860.413709] Lustre: 24354:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 860.429684] Lustre: 24354:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 913.892534] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.898838] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 913.900135] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 918.313313] Lustre: server umount lustre-OST07c6 complete [ 930.050728] Lustre: server umount lustre-MDT0000 complete [ 947.400286] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 950.745438] Key type lgssc unregistered [ 951.033823] LNet: 27874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.045538] LNetError: 27874:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.054328] LNet: Removed LNI 192.168.202.104@tcp [ 951.910700] Key type .llcrypt unregistered [ 951.913563] Key type ._llcrypt unregistered [ 972.857893] Key type ._llcrypt registered [ 972.859176] Key type .llcrypt registered [ 972.943074] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 974.003257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 974.105216] alg: No test for adler32 (adler32-zlib) [ 975.174878] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 975.378840] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 977.080165] Key type lgssc registered [ 978.365621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 986.001163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 986.016658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.757858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.333866] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 997.363285] LustreError: 29131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.404770] LustreError: 29131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1000.932915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.247459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1006.012207] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1011.951914] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1017.431446] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1026.247540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1026.711866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1032.984845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1034.070708] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1040.476978] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1063.905632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1063.917395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.930156] LustreError: Skipped 1 previous similar message [ 1063.937418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1063.950850] Lustre: Skipped 1 previous similar message [ 1063.991113] Lustre: Skipped 1 previous similar message [ 1066.452858] Lustre: server umount lustre-OST0000 complete [ 1073.121569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1073.124760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1073.157448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1074.147551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.173401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1074.176945] Lustre: Skipped 1 previous similar message [ 1077.151061] Lustre: server umount lustre-MDT0000 complete [ 1079.267248] LustreError: 29131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.296017] LustreError: 29131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1081.624188] LustreError: 29112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780852791 with bad export cookie 17908955430072367255 [ 1081.637810] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1082.217412] Lustre: server umount lustre-MDT0001 complete [ 1096.962741] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 1099.858333] Key type lgssc unregistered [ 1100.194795] LNet: 32514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.214023] LNetError: 32514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1100.226247] LNet: Removed LNI 192.168.202.104@tcp [ 1101.140538] Key type .llcrypt unregistered [ 1101.142463] Key type ._llcrypt unregistered [ 1125.067330] Key type ._llcrypt registered [ 1125.072411] Key type .llcrypt registered [ 1125.155242] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1126.549783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1126.619739] alg: No test for adler32 (adler32-zlib) [ 1127.900172] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 1128.215989] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 1129.936815] Key type lgssc registered [ 1131.314087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1140.037208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1140.060394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.671838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.340857] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1151.969824] LustreError: 33768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1151.991498] LustreError: 33768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1155.487101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1155.815469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1159.841934] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1166.402859] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1170.564699] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1177.454315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1177.839157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1184.467661] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1187.060635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1190.196914] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1201.634117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1201.648492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.670483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1202.658498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1202.659472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1202.682835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.170824] Lustre: server umount lustre-OST0000 complete [ 1212.385807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1212.395273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.423414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1212.426399] Lustre: Skipped 1 previous similar message [ 1217.727175] Lustre: server umount lustre-MDT0000 complete [ 1218.022507] LustreError: 33769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1218.034630] LustreError: 33769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1221.443397] LustreError: 33750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780852931 with bad export cookie 10790275264854730503 [ 1221.454262] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1221.454401] LustreError: 33750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1221.916074] Lustre: server umount lustre-MDT0001 complete [ 1233.684243] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 1236.788475] Key type lgssc unregistered [ 1237.121635] LNet: 36946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.132508] LNetError: 36946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.149071] LNet: Removed LNI 192.168.202.104@tcp [ 1238.152219] Key type .llcrypt unregistered [ 1238.154459] Key type ._llcrypt unregistered [ 1266.009474] Key type ._llcrypt registered [ 1266.013890] Key type .llcrypt registered [ 1266.127316] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1267.100158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1267.126938] alg: No test for adler32 (adler32-zlib) [ 1268.310760] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 1268.664240] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 1270.320151] Key type lgssc registered [ 1271.395299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1279.512906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1279.537741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.040069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.227590] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1290.227608] LustreError: 38350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.252084] LustreError: 38350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1295.331793] LustreError: 38349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.381452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.675363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1300.656446] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1307.150028] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1312.104378] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1319.302638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1319.784241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.983446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1327.330202] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1333.807584] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1340.573172] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 1341.923106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1341.934467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.968651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1342.447557] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1342.464956] Lustre: Skipped 1 previous similar message [ 1343.618993] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 1343.622954] Lustre: Skipped 3 previous similar messages [ 1346.705344] Lustre: server umount lustre-MDT0000 complete [ 1347.559103] LustreError: 39317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.587309] LustreError: 39317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1350.452518] LustreError: 38330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780853060 with bad export cookie 1220150908733724527 [ 1350.463541] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1350.476335] LustreError: 38330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1352.677867] LustreError: 39462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.684037] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.704904] LustreError: 39462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1352.742941] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1356.981678] Lustre: server umount lustre-MDT0001 complete [ 1372.165860] Lustre: server umount lustre-OST0000 complete [ 1389.948212] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 1392.579992] Key type lgssc unregistered [ 1392.833796] LNet: 41855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.839404] LNetError: 41855:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.857260] LNet: Removed LNI 192.168.202.104@tcp [ 1393.759507] Key type .llcrypt unregistered [ 1393.762121] Key type ._llcrypt unregistered [ 1418.660515] Key type ._llcrypt registered [ 1418.662920] Key type .llcrypt registered [ 1418.820412] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1420.062263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1420.200412] alg: No test for adler32 (adler32-zlib) [ 1421.205481] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 1421.394508] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 1423.008189] Key type lgssc registered [ 1424.002724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1432.524219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1432.582495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1434.321116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.469543] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1444.837174] LustreError: 43269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1444.861333] LustreError: 43269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1449.967246] LustreError: 43270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1450.623060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.150271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1456.442633] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1462.497050] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1466.237799] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1473.517033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1473.897633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1480.085777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1480.245762] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1486.876261] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1494.698543] Lustre: lustre-OST0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 1495.530838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.535768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1495.545355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1495.560987] Lustre: Skipped 1 previous similar message [ 1499.774400] Lustre: lustre-OST0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 1499.787062] Lustre: Skipped 1 previous similar message [ 1504.903436] Lustre: lustre-OST0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 1504.908771] Lustre: Skipped 2 previous similar messages [ 1507.296145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1507.553630] Lustre: server umount lustre-OST0000 complete [ 1526.241771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.244456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1526.278885] Lustre: Skipped 1 previous similar message [ 1526.293236] Lustre: Skipped 3 previous similar messages [ 1532.515683] Lustre: server umount lustre-MDT0000 complete [ 1535.461233] LustreError: 43270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.476565] LustreError: 43270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1536.916358] LustreError: 43987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780853246 with bad export cookie 7933779076742736099 [ 1536.927255] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1536.937342] LustreError: 43987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1551.332188] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1551.721151] Lustre: server umount lustre-MDT0001 complete [ 1563.380610] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 1566.262631] Key type lgssc unregistered [ 1566.619541] LNet: 46622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1566.623964] LNetError: 46622:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1566.641046] LNet: Removed LNI 192.168.202.104@tcp [ 1567.488969] Key type .llcrypt unregistered [ 1567.491172] Key type ._llcrypt unregistered [ 1594.772573] Key type ._llcrypt registered [ 1594.774985] Key type .llcrypt registered [ 1594.882454] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1596.205649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1596.309387] alg: No test for adler32 (adler32-zlib) [ 1597.406905] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 1597.615469] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 1599.312176] Key type lgssc registered [ 1600.446174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1608.856292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1608.874659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1610.345828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1615.940730] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1620.972535] LustreError: 48023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1620.995688] LustreError: 48023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1625.848185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.081359] LustreError: 48024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1626.316975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1631.312252] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1637.850348] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1642.359453] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1649.864715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1650.255528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1655.452017] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1657.753387] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1663.900555] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1670.364901] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1674.403775] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1687.190507] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-0713h29m55s [ 1688.699874] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1691.617727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1691.620225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1691.643281] Lustre: Skipped 2 previous similar messages [ 1691.663933] Lustre: Skipped 2 previous similar messages [ 1692.302165] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 1693.374329] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 1693.381178] Lustre: Skipped 3 previous similar messages [ 1697.455209] Lustre: server umount lustre-MDT0000 complete [ 1698.429607] LustreError: 48018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1701.205826] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1704.587735] LustreError: 48019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1704.615708] LustreError: 48019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1709.057232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.283943] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1709.655490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.720569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1710.725903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1715.097567] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1719.381518] Lustre: DEBUG MARKER: Wait for df (36077) ... [ 1721.229314] Lustre: DEBUG MARKER: done [ 1726.436851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.463476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1726.472150] Lustre: Skipped 1 previous similar message [ 1731.577052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1731.585451] Lustre: Skipped 1 previous similar message [ 1732.702859] Lustre: server umount lustre-OST0000 complete [ 1741.796703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.802017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1741.814458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1741.816897] Lustre: Skipped 2 previous similar messages [ 1741.841895] Lustre: Skipped 1 previous similar message [ 1743.365873] Lustre: server umount lustre-MDT0000 complete [ 1747.007639] LustreError: 50310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780853456 with bad export cookie 2064744985723622108 [ 1747.021794] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1747.032732] LustreError: 50310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1747.531737] Lustre: server umount lustre-MDT0001 complete [ 1759.869301] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 1762.492711] Key type lgssc unregistered [ 1762.765019] LNet: 52573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.770605] LNetError: 52573:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.792938] LNet: Removed LNI 192.168.202.104@tcp [ 1763.631077] Key type .llcrypt unregistered [ 1763.633300] Key type ._llcrypt unregistered [ 1808.150978] Key type ._llcrypt registered [ 1808.155522] Key type .llcrypt registered [ 1808.264992] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1809.787846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1809.878560] alg: No test for adler32 (adler32-zlib) [ 1811.073963] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 1811.422188] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 1813.176243] Key type lgssc registered [ 1814.737103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1823.847360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1823.863985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1825.214302] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1825.252329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1825.535387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.775387] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1841.259441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.594699] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1841.672657] Lustre: 54768:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1842.137328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1849.134917] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1855.282792] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1861.150764] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1868.704106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1868.895909] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1869.051433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1875.099889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1875.895160] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1882.676255] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1889.717237] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-0713h33m19s [ 1891.584565] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1899.527452] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1902.662573] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1902.757422] LustreError: 54773:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1923.239899] Lustre: lustre-MDT0000: Client c74a83b5-c554-4eab-909c-9c3e79be3109 (at 192.168.202.4@tcp) reconnecting [ 1942.848354] LustreError: 54773:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1945.991437] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1949.271559] Lustre: DEBUG MARKER: done [ 1956.833525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1956.844628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.879081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1957.345283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1957.358996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.396147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1957.412732] Lustre: Skipped 1 previous similar message [ 1961.770298] Lustre: server umount lustre-OST0000 complete [ 1967.589161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.607658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1967.617590] Lustre: Skipped 1 previous similar message [ 1974.757147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1974.769376] Lustre: Skipped 3 previous similar messages [ 1980.384153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1980.667144] Lustre: server umount lustre-MDT0000 complete [ 1985.157860] LustreError: 54759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780853695 with bad export cookie 10155247868660374474 [ 1985.163189] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1985.168719] LustreError: 54759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1985.586258] Lustre: server umount lustre-MDT0001 complete [ 2000.534631] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 2004.984863] Key type lgssc unregistered [ 2005.442792] LNet: 58745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.445639] LNetError: 58745:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2005.485863] LNet: Removed LNI 192.168.202.104@tcp [ 2006.541045] Key type .llcrypt unregistered [ 2006.543182] Key type ._llcrypt unregistered [ 2058.689337] Key type ._llcrypt registered [ 2058.691895] Key type .llcrypt registered [ 2058.800215] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2060.137070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2060.332654] alg: No test for adler32 (adler32-zlib) [ 2061.463818] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 2061.672369] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 2063.336447] Key type lgssc registered [ 2064.336945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2067.239822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2067.266558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.702643] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2068.725280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2069.028406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.159204] Lustre: Failing over lustre-MDT0000 [ 2070.828245] Lustre: server umount lustre-MDT0000 complete [ 2077.947343] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:36:27 (1780853787) [ 2085.437670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.287529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.338962] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2099.945596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.061877] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2100.120850] Lustre: 61531:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2100.483505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2105.267950] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2111.184825] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2115.431411] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2123.408696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2123.548739] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2123.832722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.635505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2130.908450] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2137.547321] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2143.571489] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2156.522344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.534720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2156.556152] Lustre: Skipped 1 previous similar message [ 2156.566584] Lustre: Skipped 1 previous similar message [ 2158.677801] Lustre: server umount lustre-MDT0000 complete [ 2162.793408] LustreError: 61521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780853872 with bad export cookie 5025990930001682908 [ 2162.810340] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2162.814346] LustreError: 61521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2163.124204] Lustre: server umount lustre-MDT0001 complete [ 2167.899942] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2173.771937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2182.117517] Lustre: 60452:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853876/real 1780853876] req@000000002f2a427d x1867360523822208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780853892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2182.136227] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.148066] Lustre: Skipped 1 previous similar message [ 2183.142251] Lustre: 60453:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853877/real 1780853877] req@00000000d7dae2a4 x1867360523822336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780853893 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2184.249804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.297444] Lustre: 60453:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853881/real 1780853881] req@000000006966a911 x1867360523822464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780853897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2187.558855] LustreError: 63152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.569582] LustreError: 63152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2187.651912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.789868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2191.652834] LustreError: 63153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.680675] LustreError: 64966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.695104] LustreError: 64966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2192.954104] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2193.903148] Lustre: 60452:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853887/real 1780853887] req@00000000b6b1221d x1867360523822848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780853903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2193.926564] Lustre: 60452:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2197.992466] LustreError: 64965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.022038] LustreError: 64965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2203.081518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.122518] LustreError: 64966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2203.141926] LustreError: 64966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2203.411184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2207.451991] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2208.752447] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2211.840489] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2214.588260] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2219.303727] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2226.155211] Lustre: server umount lustre-MDT0000 complete [ 2227.180156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.190856] LustreError: 63152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.211672] LustreError: 63152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2229.896356] LustreError: 61523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780853939 with bad export cookie 5025990930001685183 [ 2229.897578] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2229.908853] LustreError: 61523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2230.186809] Lustre: server umount lustre-MDT0001 complete [ 2237.278936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2247.092501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.672427] Lustre: 60451:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853942/real 1780853942] req@00000000647a8100 x1867360523847552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780853958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2248.709678] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.731721] Lustre: Skipped 2 previous similar messages [ 2254.882665] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x45bfe81f3b15f022 [ 2254.924945] Lustre: MGC192.168.202.104@tcp: Connection restored to 0@lo (at 0@lo) [ 2255.202305] LustreError: 63152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.328272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2255.418035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2259.701144] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2260.449547] Lustre: 60451:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853954/real 1780853954] req@00000000791899ab x1867360523848192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780853970 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2260.472458] Lustre: 60451:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2267.637961] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2268.510817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2274.181279] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2274.284876] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2279.466108] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2283.861866] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2290.812743] Lustre: server umount lustre-MDT0000 complete [ 2294.242150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2294.249080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.256784] LustreError: 63153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.261900] LustreError: 63153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2294.847466] LustreError: 61522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780854004 with bad export cookie 5025990930001686562 [ 2294.852790] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2294.858800] LustreError: 61522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2295.207656] Lustre: server umount lustre-MDT0001 complete [ 2301.922856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2305.192776] Lustre: DEBUG MARKER: files compared the same [ 2317.619491] Lustre: server umount lustre-OST0000 complete [ 2336.301815] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 2339.265525] Key type lgssc unregistered [ 2339.534221] LNet: 70452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.545315] LNetError: 70452:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.576735] LNet: Removed LNI 192.168.202.104@tcp [ 2340.523904] Key type .llcrypt unregistered [ 2340.526820] Key type ._llcrypt unregistered [ 2366.436790] Key type ._llcrypt registered [ 2366.440165] Key type .llcrypt registered [ 2366.553863] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2367.387849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2367.419554] alg: No test for adler32 (adler32-zlib) [ 2368.517699] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 2368.714770] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 2370.376167] Key type lgssc registered [ 2371.341919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2378.039283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2378.058384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.619053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.163151] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug +malloc all [ 2389.991257] LustreError: 71853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.023029] LustreError: 71853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2392.401528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2392.669381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2396.576763] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug +malloc all [ 2402.078989] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2406.539777] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2413.211651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2413.534449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2419.156790] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug +malloc all [ 2424.816140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2425.011612] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2435.043669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2435.048259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.066454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2435.074681] Lustre: Skipped 1 previous similar message [ 2435.096821] Lustre: Skipped 1 previous similar message [ 2439.725705] Lustre: server umount lustre-OST0000 complete [ 2443.745070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2443.750526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2443.759887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2445.281100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2445.282068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.656430] Lustre: server umount lustre-MDT0000 complete [ 2450.402407] LustreError: 71853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.427302] LustreError: 71853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2453.279867] LustreError: 71834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780854163 with bad export cookie 14390509435613241125 [ 2453.287947] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2453.593714] Lustre: server umount lustre-MDT0001 complete [ 2466.147846] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 2468.337384] Key type lgssc unregistered [ 2468.567395] LNet: 75026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2468.579933] LNetError: 75026:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2468.599533] LNet: Removed LNI 192.168.202.104@tcp [ 2469.249583] Key type .llcrypt unregistered [ 2469.251551] Key type ._llcrypt unregistered [ 2490.743440] Key type ._llcrypt registered [ 2490.745726] Key type .llcrypt registered [ 2490.851708] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2491.913800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2491.925720] alg: No test for adler32 (adler32-zlib) [ 2492.976252] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 2493.253346] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 2494.952140] Key type lgssc registered [ 2495.914074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2503.517680] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2503.547129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2519.072391] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2524.192199] LustreError: 76401:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.104@tcp: failed processing log, type 4: rc = -110 [ 2550.880231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.706814] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2564.459087] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2573.426792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.580788] Lustre: *** cfs_fail_loc=706, val=0*** [ 2573.924397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2574.016145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2578.603371] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2578.920630] LustreError: 76426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.954084] LustreError: 76426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2584.044517] LustreError: 76427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.124951] LustreError: 76426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.136810] LustreError: 76426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2587.957379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.444818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2593.582250] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2599.162882] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2603.068029] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2609.122686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2609.126748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.133662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2609.144390] Lustre: Skipped 1 previous similar message [ 2614.249474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2614.255353] Lustre: Skipped 1 previous similar message [ 2619.361251] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2619.372169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2619.386821] Lustre: Skipped 2 previous similar messages [ 2619.623898] Lustre: server umount lustre-OST0000 complete [ 2624.482489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2624.488178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2624.490485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2629.606666] LustreError: 77375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.619832] LustreError: 77375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2629.803604] Lustre: server umount lustre-MDT0000 complete [ 2633.442369] LustreError: 78102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780854343 with bad export cookie 16602635588483891906 [ 2633.451248] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2633.456663] LustreError: 78102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2633.922334] Lustre: server umount lustre-MDT0001 complete [ 2643.969426] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 2646.581651] Key type lgssc unregistered [ 2646.780868] LNet: 79662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2646.785894] LNetError: 79662:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2647.846931] LNet: Removed LNI 192.168.202.104@tcp [ 2648.525193] Key type .llcrypt unregistered [ 2648.532541] Key type ._llcrypt unregistered [ 2673.833447] Key type ._llcrypt registered [ 2673.835768] Key type .llcrypt registered [ 2673.927672] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2675.119191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2675.157197] alg: No test for adler32 (adler32-zlib) [ 2676.278167] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 2676.602096] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 2678.336124] Key type lgssc registered [ 2679.580864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2687.462062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2687.484676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2692.786567] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2700.038861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2700.419445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2704.618961] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2705.890587] LustreError: 81735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.917217] LustreError: 81735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2711.016390] LustreError: 81739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.026804] LustreError: 81739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2712.865953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2713.339638] LustreError: 81735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2713.369849] LustreError: 81735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2713.401203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2718.701601] LustreError: 81734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.722771] LustreError: 81734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2719.934749] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2723.815268] LustreError: 81734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.833719] LustreError: 81734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2729.965896] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2729.979047] Lustre: MGS: Client dbfb21da-d89a-4828-9250-723249bc714e (at 0@lo) reconnecting [ 2730.052637] LustreError: 81749:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2730.306699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2730.411133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2737.070408] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2756.071087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.078179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2756.079406] Lustre: Skipped 1 previous similar message [ 2756.080537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2756.089327] Lustre: Skipped 1 previous similar message [ 2758.439987] Lustre: server umount lustre-OST0000 complete [ 2766.309970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.319581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2766.320258] Lustre: Skipped 1 previous similar message [ 2766.321145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2766.348169] Lustre: Skipped 1 previous similar message [ 2768.750917] Lustre: server umount lustre-MDT0000 complete [ 2771.425883] LustreError: 81739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.454466] LustreError: 81739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2773.394111] Lustre: server umount lustre-MDT0001 complete [ 2777.351982] Lustre: server umount MGS complete [ 2791.856741] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:48:21 (1780854501) [ 2825.201778] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 2836.721355] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2871.979357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.901379] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2885.085757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.281641] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2885.314187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2885.371668] Lustre: lustre-MDT0001: new disk, initializing [ 2885.459960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2885.501174] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2885.507104] Lustre: 88822:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2885.512171] LustreError: 88822:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2889.749932] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2900.832461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2901.078738] Lustre: lustre-OST0000: new disk, initializing [ 2901.082133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2901.087124] Lustre: 89571:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2901.181247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.566918] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2909.735674] Lustre: 89913:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000d7dae2a4 x1867361168696320/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780854635 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2916.895291] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2923.874986] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2923.899143] Lustre: MGS: Client 3eda16c3-d3bb-4eaf-8022-1dad72dc4176 (at 0@lo) reconnecting [ 2923.964635] Lustre: 90505:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2923.986886] LustreError: 88808:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2923.995831] LustreError: 88808:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2924.018455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2924.069944] Lustre: lustre-MDT0000: new disk, initializing [ 2924.170901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.191600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2924.209855] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2925.281280] Lustre: 90529:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000000027fc3 x1867361168702720/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1780854651 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2929.123914] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2930.472658] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2934.386541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2934.419807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2934.557603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2962.923815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2962.931854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2962.935206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2962.945940] Lustre: Skipped 1 previous similar message [ 2966.748492] Lustre: server umount lustre-OST0000 complete [ 2973.155725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.175314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2973.182074] Lustre: Skipped 2 previous similar messages [ 2976.777187] Lustre: server umount lustre-MDT0000 complete [ 2978.277683] LustreError: 88793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2978.294068] LustreError: 88793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2995.168184] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2995.673915] Lustre: server umount lustre-MDT0001 complete [ 3000.254714] Lustre: server umount MGS complete [ 3013.176136] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:52:02 (1780854722) [ 3031.898995] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3034.280780] Key type lgssc unregistered [ 3034.592638] LNet: 92998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3034.596369] LNetError: 92998:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3034.611657] LNet: Removed LNI 192.168.202.104@tcp [ 3035.330345] Key type .llcrypt unregistered [ 3035.332344] Key type ._llcrypt unregistered [ 3053.941519] Key type ._llcrypt registered [ 3053.946290] Key type .llcrypt registered [ 3054.086237] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3055.877602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3056.059919] alg: No test for adler32 (adler32-zlib) [ 3057.419707] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 3057.724417] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3059.496211] Key type lgssc registered [ 3061.392752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3071.025989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3071.069054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.623029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.824885] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3082.220564] LustreError: 94203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.238662] LustreError: 94203:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3085.667124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.044967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3090.370671] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3096.199308] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3100.620683] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3107.206643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3107.599313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.810575] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3117.875904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3120.166572] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3131.365032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3131.374857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3131.390559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3133.411243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3133.411370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.686215] Lustre: server umount lustre-OST0000 complete [ 3142.625239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3142.635131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.646083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3142.649580] Lustre: Skipped 1 previous similar message [ 3145.890385] Lustre: server umount lustre-MDT0000 complete [ 3148.774980] LustreError: 94198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.792847] LustreError: 94198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3149.520151] LustreError: 94920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780854859 with bad export cookie 17158748960576164533 [ 3149.526268] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3149.544553] LustreError: 94920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3149.898511] Lustre: server umount lustre-MDT0001 complete [ 3161.914361] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3164.663836] Key type lgssc unregistered [ 3165.004595] LNet: 97377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3165.011784] LNetError: 97377:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3165.034391] LNet: Removed LNI 192.168.202.104@tcp [ 3165.878680] Key type .llcrypt unregistered [ 3165.880785] Key type ._llcrypt unregistered [ 3182.281174] Key type ._llcrypt registered [ 3182.284225] Key type .llcrypt registered [ 3182.408136] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3183.513658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3183.547612] alg: No test for adler32 (adler32-zlib) [ 3184.592848] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 3184.768951] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3186.488583] Key type lgssc registered [ 3187.757148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3193.501384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3193.509665] LustreError: 98447:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3193.521301] LustreError: 98450:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3193.528668] LustreError: 98447:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 3193.536445] LustreError: 98447:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3193.540485] LustreError: 98447:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3193.545695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.995772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3202.159557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3202.457596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3206.774811] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3211.568707] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing lsmod [ 3214.312839] LustreError: 99618:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3214.338812] LustreError: 99622:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3214.344269] LustreError: 99618:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4982 [ 3214.351679] LustreError: 99618:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3214.361409] LustreError: 99618:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3214.365503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3214.760726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.890829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3217.899283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.916424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3219.937911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3219.939706] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.953778] Lustre: Skipped 1 previous similar message [ 3223.610669] Lustre: server umount lustre-MDT0000 complete [ 3225.062518] LustreError: 99034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.079084] LustreError: 99034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3227.120907] LustreError: 98489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3227.148327] LustreError: 98489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3227.381234] LustreError: 98465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780854937 with bad export cookie 13988001109895456484 [ 3227.385930] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3227.396838] LustreError: 98465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3229.280233] LustreError: 98919:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000052955ebc x1867361701350656/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3229.487932] Lustre: server umount lustre-MDT0001 complete [ 3243.982659] Lustre: server umount lustre-OST0000 complete [ 3253.622324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.223285] LustreError: 100687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.320955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3259.365462] LustreError: 100688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3259.571759] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3264.489872] LustreError: 100687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3267.588855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3268.041312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3271.982720] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3277.438792] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3281.454191] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3287.830993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3288.125948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3294.097495] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3299.395136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3299.953535] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3307.490583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3307.493050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3307.507526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3307.518457] Lustre: Skipped 2 previous similar messages [ 3309.538382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.542806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3314.663244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3314.681865] Lustre: Skipped 2 previous similar messages [ 3321.313740] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3321.613794] Lustre: server umount lustre-OST0000 complete [ 3329.507067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3329.514760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3329.563655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3329.574407] Lustre: Skipped 2 previous similar messages [ 3341.280626] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3341.551566] Lustre: server umount lustre-MDT0000 complete [ 3345.378794] LustreError: 100688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.414254] LustreError: 100688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3347.236484] LustreError: 100669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780855057 with bad export cookie 13988001109895457163 [ 3347.246987] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3347.266934] LustreError: 100669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3347.727789] Lustre: server umount lustre-MDT0001 complete [ 3361.753619] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3365.687884] Key type lgssc unregistered [ 3366.010737] LNet: 103866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3366.022639] LNetError: 103866:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3366.043308] LNet: Removed LNI 192.168.202.104@tcp [ 3367.044061] Key type .llcrypt unregistered [ 3367.045664] Key type ._llcrypt unregistered [ 3395.231327] Key type ._llcrypt registered [ 3395.233250] Key type .llcrypt registered [ 3395.305498] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3396.453074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3396.479445] alg: No test for adler32 (adler32-zlib) [ 3397.802557] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 3398.088034] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3399.808129] Key type lgssc registered [ 3401.134307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3409.308611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3409.350522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.840925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3415.652436] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3420.138262] LustreError: 105266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.164701] LustreError: 105266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3424.585176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3424.969382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3428.496960] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3433.903862] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3438.524528] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3444.754393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3445.168209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3450.738241] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3455.424740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3457.131673] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3462.247334] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3466.602887] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3474.916404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3474.931481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.957156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3475.937448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3475.943349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3475.951404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3475.977104] Lustre: Skipped 1 previous similar message [ 3481.068249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3481.081494] Lustre: Skipped 1 previous similar message [ 3486.176543] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3486.187459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3486.197122] Lustre: Skipped 1 previous similar message [ 3486.568326] Lustre: server umount lustre-OST0000 complete [ 3491.810789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3491.829254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3491.863816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3496.420210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3497.725763] Lustre: server umount lustre-MDT0000 complete [ 3501.560967] LustreError: 107358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3501.598565] LustreError: 107358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3501.652056] LustreError: 105985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780855211 with bad export cookie 9540363177012442953 [ 3501.660615] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3501.669516] LustreError: 105985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3502.019288] Lustre: server umount lustre-MDT0001 complete [ 3514.227668] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3516.915464] Key type lgssc unregistered [ 3517.222780] LNet: 108549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3517.230034] LNetError: 108549:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3517.245838] LNet: Removed LNI 192.168.202.104@tcp [ 3518.007983] Key type .llcrypt unregistered [ 3518.010202] Key type ._llcrypt unregistered [ 3536.867368] Key type ._llcrypt registered [ 3536.869183] Key type .llcrypt registered [ 3536.974446] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3538.139139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3538.155701] alg: No test for adler32 (adler32-zlib) [ 3539.300252] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 3539.574826] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3541.240200] Key type lgssc registered [ 3542.487601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3549.987324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3550.000368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3551.589196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3555.871796] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3561.956351] LustreError: 109755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3561.971773] LustreError: 109755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3563.704498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.040094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3569.291600] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3575.064686] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3578.662131] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3585.570258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3585.969704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.783197] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3596.284926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3598.148779] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3609.571560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3609.575136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3609.594134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3611.621150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3611.621149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3611.648437] Lustre: Skipped 1 previous similar message [ 3613.393100] Lustre: server umount lustre-OST0000 complete [ 3620.321573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3620.340562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.353703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3623.235328] Lustre: server umount lustre-MDT0000 complete [ 3626.983139] LustreError: 109755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3627.001886] LustreError: 109755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3627.121906] LustreError: 109736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780855337 with bad export cookie 4714451034552274700 [ 3627.124758] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3627.134320] LustreError: 109736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3627.453297] Lustre: server umount lustre-MDT0001 complete [ 3638.139487] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3640.717210] Key type lgssc unregistered [ 3640.968941] LNet: 112944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3640.979706] LNetError: 112944:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3642.028948] LNet: Removed LNI 192.168.202.104@tcp [ 3642.873404] Key type .llcrypt unregistered [ 3642.877404] Key type ._llcrypt unregistered [ 3668.835868] Key type ._llcrypt registered [ 3668.840460] Key type .llcrypt registered [ 3668.984145] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3670.033610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3670.196532] alg: No test for adler32 (adler32-zlib) [ 3671.214944] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 3671.527174] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3673.225129] Key type lgssc registered [ 3674.503209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3682.885122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3682.894126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.359046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3688.686717] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3694.566554] LustreError: 114356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3694.583471] LustreError: 114356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3696.927692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.257277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3701.344331] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3707.358384] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3710.813803] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3718.370788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3718.812667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3726.361165] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3730.096824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3731.820842] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3739.876207] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3744.071595] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3744.075841] Lustre: Skipped 2 previous similar messages [ 3748.290701] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3748.297581] Lustre: Skipped 2 previous similar messages [ 3749.402741] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3749.440235] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3759.464201] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.4@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3759.473272] Lustre: Skipped 2 previous similar messages [ 3768.354932] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.4@tcp 192.168.0.[2,10]@tcp [ 3772.312363] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.4/24@tcp in log lustre-MDT0000 [ 3772.318139] Lustre: Skipped 2 previous similar messages [ 3774.235647] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.4/24@tcp [ 3774.254817] Lustre: Skipped 1 previous similar message [ 3779.555484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3779.563448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3779.565937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3779.577060] Lustre: Skipped 1 previous similar message [ 3781.887977] Lustre: server umount lustre-OST0000 complete [ 3789.282283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3789.289919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.307640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3789.319846] Lustre: Skipped 1 previous similar message [ 3791.572755] Lustre: server umount lustre-MDT0000 complete [ 3794.760143] LustreError: 114336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780855504 with bad export cookie 2883451351198007437 [ 3794.766426] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3794.769728] LustreError: 114336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3795.039827] Lustre: server umount lustre-MDT0001 complete [ 3803.622530] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3805.804644] Key type lgssc unregistered [ 3806.072753] LNet: 118882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3806.079604] LNetError: 118882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3806.092784] LNet: Removed LNI 192.168.202.104@tcp [ 3806.653786] Key type .llcrypt unregistered [ 3806.656100] Key type ._llcrypt unregistered [ 3829.399504] Key type ._llcrypt registered [ 3829.401719] Key type .llcrypt registered [ 3829.496270] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3830.533549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3830.561553] alg: No test for adler32 (adler32-zlib) [ 3831.536558] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 3831.727921] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3833.385339] Key type lgssc registered [ 3834.338781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3840.379986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3840.393645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3841.712845] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3841.732541] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.4/24@tcp [ 3841.752357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.504561] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3851.235989] LustreError: 120405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3851.250955] LustreError: 120405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3853.192218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.473239] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3853.509977] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.4/24@tcp [ 3853.557749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3856.966981] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3861.841470] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3865.666952] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3871.988819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3872.424987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3877.298588] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3882.729833] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3886.837968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3897.314848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.323618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3897.323729] Lustre: Skipped 1 previous similar message [ 3897.336376] Lustre: Skipped 1 previous similar message [ 3902.435435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3902.993268] Lustre: server umount lustre-OST0000 complete [ 3907.559236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.581137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3907.587131] Lustre: Skipped 2 previous similar messages [ 3912.426176] Lustre: server umount lustre-MDT0000 complete [ 3912.679095] LustreError: 120401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3912.696906] LustreError: 120401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3915.876626] LustreError: 120385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780855625 with bad export cookie 13351366572174295274 [ 3915.878103] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3915.890363] LustreError: 120385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3916.158241] Lustre: server umount lustre-MDT0001 complete [ 3926.063099] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3928.498923] Key type lgssc unregistered [ 3928.700929] LNet: 123635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3928.708909] LNetError: 123635:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3929.768285] LNet: Removed LNI 192.168.202.104@tcp [ 3930.545319] Key type .llcrypt unregistered [ 3930.547199] Key type ._llcrypt unregistered [ 3952.758189] Key type ._llcrypt registered [ 3952.762390] Key type .llcrypt registered [ 3952.877802] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 3966.157895] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3967.430885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3967.458216] alg: No test for adler32 (adler32-zlib) [ 3968.418780] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 3968.587612] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3970.257052] Key type lgssc registered [ 3971.324074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4000.072937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4000.101289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.373780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4001.394410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4001.489743] Lustre: lustre-MDT0000: new disk, initializing [ 4001.593540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4001.632749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4005.822755] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4015.340606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4015.487031] Lustre: 127001:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4015.543799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4015.547606] Lustre: Skipped 1 previous similar message [ 4015.619499] Lustre: lustre-MDT0001: new disk, initializing [ 4015.709286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4015.756692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4015.766265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4020.405778] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4027.731481] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4031.240911] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4037.580732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4037.901403] Lustre: lustre-OST0000: new disk, initializing [ 4037.906419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4037.913766] Lustre: 128741:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4038.000363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4042.918970] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4044.840836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4044.846380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4044.922219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4049.400755] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4053.088511] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4053.282707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4056.200967] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4056.356289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4060.133677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4060.135257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4060.146559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4060.154986] Lustre: Skipped 1 previous similar message [ 4064.157958] Lustre: server umount lustre-OST0000 complete [ 4070.376182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4070.376948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4070.397837] Lustre: Skipped 2 previous similar messages [ 4073.343883] Lustre: server umount lustre-MDT0000 complete [ 4076.187471] LustreError: 126993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780855786 with bad export cookie 16455170443967911954 [ 4076.188768] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4076.199617] LustreError: 126993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4076.430702] Lustre: server umount lustre-MDT0001 complete [ 4082.878217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4083.195632] LustreError: 130809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4083.208744] LustreError: 130809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4083.245688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4086.305973] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4088.292556] LustreError: 130810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4091.747702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4091.975849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4094.968747] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4099.304434] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4102.466675] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4107.645749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4107.901070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.073803] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4117.256868] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4120.325920] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4120.482729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4123.312365] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4123.451132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4130.459879] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4130.624964] Lustre: lustre-OST0001: new disk, initializing [ 4130.629100] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4130.634850] Lustre: 133805:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4130.725973] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4135.710775] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4140.150295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4140.166538] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4140.291543] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4142.790739] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4142.965516] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4146.868542] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4147.025907] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4150.036521] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff940e8400a800.ost_server_uuid 50 [ 4151.331663] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff940e8400a800.ost_server_uuid in FULL state after 0 sec [ 4160.994266] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4161.000438] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4161.013588] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4161.020051] Lustre: Skipped 2 previous similar messages [ 4165.062968] Lustre: server umount lustre-OST0001 complete [ 4166.117181] LustreError: 132402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4166.131772] LustreError: 132402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4170.723601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4170.744677] LustreError: Skipped 1 previous similar message [ 4170.755701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4170.766460] Lustre: Skipped 1 previous similar message [ 4170.773517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4171.240275] LustreError: 132404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.250559] LustreError: 132404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4174.088167] Lustre: server umount lustre-OST0000 complete [ 4178.913332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4178.916879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4178.927731] Lustre: Skipped 1 previous similar message [ 4178.938628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4178.947119] Lustre: Skipped 2 previous similar messages [ 4183.505036] Lustre: server umount lustre-MDT0000 complete [ 4186.450567] LustreError: 130790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780855896 with bad export cookie 16455170443967912836 [ 4186.451246] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4186.460164] LustreError: 130790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4186.687846] Lustre: server umount lustre-MDT0001 complete [ 4201.267497] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 4203.575252] Key type lgssc unregistered [ 4203.855694] LNet: 136639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4203.862596] LNetError: 136639:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4203.884921] LNet: Removed LNI 192.168.202.104@tcp [ 4204.540850] Key type .llcrypt unregistered [ 4204.542367] Key type ._llcrypt unregistered [ 4235.968916] Key type ._llcrypt registered [ 4235.971607] Key type .llcrypt registered [ 4236.044632] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 4246.878927] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4247.724609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4247.737439] alg: No test for adler32 (adler32-zlib) [ 4248.770304] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 4248.978347] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 4250.616800] Key type lgssc registered [ 4251.438792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4278.926345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4278.936917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4280.102539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4280.122654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4280.188338] Lustre: lustre-MDT0000: new disk, initializing [ 4280.250582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.265491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4282.933793] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4290.353320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4290.429862] Lustre: 140738:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4290.477766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4290.481414] Lustre: Skipped 1 previous similar message [ 4290.536753] Lustre: lustre-MDT0001: new disk, initializing [ 4290.614348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4290.650474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4290.659429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4293.248765] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4299.023900] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4301.709202] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4306.182285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4306.397989] Lustre: lustre-OST0000: new disk, initializing [ 4306.401843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4306.406826] Lustre: 142479:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4306.477792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4308.041926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4308.050593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4308.149769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4310.444108] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4315.625408] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4318.774034] Lustre: Failing over lustre-OST0000 [ 4318.873842] Lustre: server umount lustre-OST0000 complete [ 4335.004089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4335.265404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4335.277704] Lustre: Skipped 1 previous similar message [ 4335.292727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4335.312569] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4337.378309] Lustre: 138965:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780856031/real 1780856031] req@000000006aa7663e x1867362817045504/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1780856047 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4337.829261] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4339.140978] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4339.344583] Lustre: 138964:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780856033/real 1780856033] req@00000000a92e76c9 x1867362817046016/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780856049 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4340.736661] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4340.741635] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4341.854336] Lustre: Failing over lustre-MDT0000 [ 4342.029539] Lustre: server umount lustre-MDT0000 complete [ 4344.928502] Lustre: 138964:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780856038/real 1780856038] req@000000002a1d4671 x1867362817047552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780856054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4344.953954] Lustre: 138964:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4345.828712] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.843040] Lustre: Skipped 2 previous similar messages [ 4348.029559] LustreError: 142486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4348.045098] LustreError: 142486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4349.920222] Lustre: 138965:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780856043/real 1780856043] req@000000001289d1ca x1867362817048576/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780856059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4350.953597] LustreError: 140745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.968504] LustreError: 140745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4353.153467] LustreError: 142486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.075362] LustreError: 143286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.093029] LustreError: 143286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4358.356552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4358.480795] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4358.492563] Lustre: 140742:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.202.104@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4358.695156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4358.750864] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4358.813349] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4361.349626] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4363.755458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4363.760017] Lustre: Skipped 1 previous similar message [ 4363.781661] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4363.831067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4368.877414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.891351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4368.896476] Lustre: Skipped 1 previous similar message [ 4373.313213] Lustre: server umount lustre-OST0000 complete [ 4379.107120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4379.115291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4379.119602] Lustre: Skipped 1 previous similar message [ 4379.125608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4379.129404] Lustre: Skipped 1 previous similar message [ 4381.836524] Lustre: server umount lustre-MDT0000 complete [ 4384.229248] LustreError: 140750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4384.243155] LustreError: 140750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4384.487522] LustreError: 140732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780856094 with bad export cookie 4040603978699064822 [ 4384.491065] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4384.495891] LustreError: 140732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4384.766409] Lustre: server umount lustre-MDT0001 complete [ 4391.431713] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 4393.159712] Key type lgssc unregistered [ 4393.374649] LNet: 145964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4393.381252] LNetError: 145964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4393.399291] LNet: Removed LNI 192.168.202.104@tcp [ 4394.002353] Key type .llcrypt unregistered [ 4394.005382] Key type ._llcrypt unregistered [ 4421.093363] Key type ._llcrypt registered [ 4421.096728] Key type .llcrypt registered [ 4421.171228] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4421.921931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4421.981750] alg: No test for adler32 (adler32-zlib) [ 4422.973808] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 4423.191146] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 4424.856371] Key type lgssc registered [ 4425.681700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4430.958835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4430.970631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.234108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4434.596992] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4439.740825] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4442.595072] LustreError: 147464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4442.610742] LustreError: 147464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4442.934692] LustreError: 147493:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4442.939486] LustreError: 147493:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4442.941912] LustreError: 147493:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 4443.157270] Lustre: server umount lustre-MDT0000 complete [ 4452.828393] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 4458.938535] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4481.633858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4481.795541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4481.824460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4481.886440] Lustre: lustre-MDT0000: new disk, initializing [ 4481.938196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.948644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4484.865988] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4492.408614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4492.487672] Lustre: 150645:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4492.524222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4492.527739] Lustre: Skipped 1 previous similar message [ 4492.570868] Lustre: lustre-MDT0001: new disk, initializing [ 4492.630699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4492.655627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4492.669280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4495.654435] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4501.462265] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4504.460588] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4509.304083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4509.500567] Lustre: lustre-OST0000: new disk, initializing [ 4509.503593] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4509.508703] Lustre: 152385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4509.562323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4513.608886] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4516.905826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4516.926605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4517.009194] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4519.936317] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4522.888884] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4524.109139] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:17:14 (1780856234) [ 4717.028613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4717.028935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4717.039409] Lustre: Skipped 1 previous similar message [ 4717.045833] Lustre: Skipped 1 previous similar message [ 4722.145575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4722.149855] Lustre: Skipped 2 previous similar messages [ 4727.265981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4727.272189] Lustre: Skipped 2 previous similar messages [ 4731.360108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4731.473508] Lustre: server umount lustre-MDT0000 complete [ 4732.385120] LustreError: 152998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4732.399100] LustreError: 152998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4733.886117] LustreError: 150638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780856443 with bad export cookie 3704013529043069154 [ 4733.887608] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4733.894819] LustreError: 150638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4737.504580] LustreError: 150653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4737.504708] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4737.532975] Lustre: Skipped 1 previous similar message [ 4737.544208] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4743.656320] LustreError: 150653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4743.657537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4743.675341] LustreError: 150653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4743.679859] Lustre: Skipped 1 previous similar message [ 4748.257069] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4748.393597] Lustre: server umount lustre-MDT0001 complete [ 4757.469248] Lustre: server umount lustre-OST0000 complete [ 4763.707700] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 4769.044549] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4792.738784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4792.980408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4793.001715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4793.056708] Lustre: lustre-MDT0000: new disk, initializing [ 4793.136267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4793.153260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4795.809838] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4802.766607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4802.846748] Lustre: 156890:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4802.893889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4802.897923] Lustre: Skipped 1 previous similar message [ 4802.951923] Lustre: lustre-MDT0001: new disk, initializing [ 4803.009559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4803.033068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4803.040695] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4804.998674] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4809.786607] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4812.079881] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4816.830683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4817.023851] Lustre: lustre-OST0000: new disk, initializing [ 4817.027112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4817.031711] Lustre: 158629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4817.224969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4817.234094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4817.307622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4820.616310] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4825.380873] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4828.056830] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4828.202420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4830.765200] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4830.919626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4832.737682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4832.739729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4832.745535] LustreError: Skipped 1 previous similar message [ 4832.746340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4832.757942] Lustre: Skipped 1 previous similar message [ 4838.145291] Lustre: server umount lustre-OST0000 complete [ 4842.983047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4846.627792] Lustre: server umount lustre-MDT0000 complete [ 4848.097578] LustreError: 156925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.107609] LustreError: 156925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4848.860155] LustreError: 156882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780856558 with bad export cookie 3704013529043227984 [ 4848.863766] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4848.868909] LustreError: 156882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4849.095364] Lustre: server umount lustre-MDT0001 complete [ 4854.216223] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:22:44 (1780856564) [ 4864.115728] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 4869.090960] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4890.620939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.799541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4890.824828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4890.881764] Lustre: lustre-MDT0000: new disk, initializing [ 4890.941699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4890.945814] Lustre: Skipped 1 previous similar message [ 4890.956608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4893.454340] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4901.237724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4901.331483] Lustre: 163197:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4901.490179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4904.122274] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4909.483647] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4911.872342] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4915.915452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4916.103683] Lustre: 164948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4919.310196] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4924.094286] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4924.439315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4924.442503] Lustre: Skipped 1 previous similar message [ 4924.447815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4924.513277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4934.625992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4934.631535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4934.632499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4934.642275] Lustre: Skipped 2 previous similar messages [ 4934.649226] Lustre: Skipped 6 previous similar messages [ 4938.985458] Lustre: server umount lustre-OST0000 complete [ 4942.305415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4942.316596] LustreError: Skipped 1 previous similar message [ 4944.870171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4944.877441] Lustre: Skipped 1 previous similar message [ 4947.507829] Lustre: server umount lustre-MDT0000 complete [ 4949.989565] LustreError: 163209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4949.997493] LustreError: 163209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4950.132966] LustreError: 163191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780856660 with bad export cookie 3704013529043228866 [ 4950.138175] LustreError: 163191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4950.138398] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4950.412468] Lustre: server umount lustre-MDT0001 complete [ 4955.283679] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:24:25 (1780856665) [ 4963.999735] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 4968.698946] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4990.398449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4990.573570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4990.594695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4990.597597] Lustre: Skipped 3 previous similar messages [ 4990.643634] Lustre: lustre-MDT0000: new disk, initializing [ 4990.645199] Lustre: Skipped 2 previous similar messages [ 4990.688282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4990.691321] Lustre: Skipped 2 previous similar messages [ 4990.702791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4993.085800] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5000.355977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5000.441655] Lustre: 169116:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5000.603409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5003.235441] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5008.280383] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5010.511233] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5014.212637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5014.353175] Lustre: 170865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5017.265700] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5022.172415] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5022.269464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5027.298476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5027.301480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5027.301893] LustreError: Skipped 1 previous similar message [ 5027.313667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5027.317582] Lustre: Skipped 3 previous similar messages [ 5031.718648] Lustre: server umount lustre-OST0000 complete [ 5036.514258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5040.421410] Lustre: server umount lustre-MDT0000 complete [ 5042.662236] LustreError: 169123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5042.680292] LustreError: 169123:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5043.002074] LustreError: 169109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780856752 with bad export cookie 3704013529043229951 [ 5043.004200] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5043.015760] LustreError: 169109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5043.267736] Lustre: server umount lustre-MDT0001 complete [ 5049.446524] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5050.934640] Key type lgssc unregistered [ 5051.145549] LNet: 172588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5051.150777] LNetError: 172588:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5052.198742] LNet: Removed LNI 192.168.202.104@tcp [ 5052.724080] Key type .llcrypt unregistered [ 5052.725512] Key type ._llcrypt unregistered [ 5066.220848] Key type ._llcrypt registered [ 5066.222659] Key type .llcrypt registered [ 5066.278471] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5066.838543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5066.898718] alg: No test for adler32 (adler32-zlib) [ 5067.790136] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 5067.897462] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 5069.512651] Key type lgssc registered [ 5070.166622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5074.622530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5074.633258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5075.874527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5077.975808] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5083.041216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5083.332575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5085.946772] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5089.654724] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5091.723834] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5095.415383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5095.600554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5098.743371] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5100.723522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5102.080266] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5126.627099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5126.636275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5126.648271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5129.703075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5129.705161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5129.707818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5130.969282] Lustre: server umount lustre-OST0000 complete [ 5134.818744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5134.826436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5134.826596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5134.831343] Lustre: Skipped 1 previous similar message [ 5134.841827] Lustre: Skipped 2 previous similar messages [ 5138.707659] Lustre: server umount lustre-MDT0000 complete [ 5139.937627] LustreError: 176114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5139.947494] LustreError: 176114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5140.617224] LustreError: 173943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780856850 with bad export cookie 8999194622431207996 [ 5140.620604] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5140.622249] LustreError: 173943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5140.802041] Lustre: server umount lustre-MDT0001 complete [ 5145.985815] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5147.507931] Key type lgssc unregistered [ 5147.694066] LNet: 177159:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5147.698574] LNetError: 177159:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5147.709392] LNet: Removed LNI 192.168.202.104@tcp [ 5148.147674] Key type .llcrypt unregistered [ 5148.149522] Key type ._llcrypt unregistered [ 5160.309540] Key type ._llcrypt registered [ 5160.311780] Key type .llcrypt registered [ 5160.361383] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5160.746822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5160.805098] alg: No test for adler32 (adler32-zlib) [ 5161.731189] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 5161.868195] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 5163.496204] Key type lgssc registered [ 5164.145702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5167.941623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5167.953230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5169.240608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5171.359791] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5176.063722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5176.287752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5178.607032] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5181.820907] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5183.822102] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5187.388590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5187.567391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5190.917848] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5193.863735] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5194.737817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5207.011764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5207.012595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5207.017491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5207.026766] Lustre: Skipped 1 previous similar message [ 5209.007302] Lustre: server umount lustre-OST0000 complete [ 5211.778268] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff940e88c08000.ost_server_uuid 50 [ 5227.273137] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5237.729097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5237.730354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.733779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5237.739966] Lustre: Skipped 1 previous similar message [ 5237.744310] Lustre: Skipped 2 previous similar messages [ 5242.851377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5242.856159] Lustre: Skipped 1 previous similar message [ 5243.817042] Lustre: server umount lustre-MDT0000 complete [ 5245.375378] LustreError: 178518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780856955 with bad export cookie 3511212596225207284 [ 5245.376795] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5245.379347] LustreError: 178518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5245.512339] Lustre: server umount lustre-MDT0001 complete [ 5249.064353] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:29:19 (1780856959) [ 5252.004512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.248711] LustreError: 182036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5252.257042] LustreError: 182036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5252.277442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5253.807229] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5257.650908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.698309] LustreError: 182037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5257.823739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5259.474781] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5261.940885] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5263.516622] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5266.526841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5266.767146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5269.436395] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5271.810249] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5275.110429] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5275.210471] Lustre: lustre-OST0001: new disk, initializing [ 5275.213337] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5275.217066] Lustre: 184533:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5277.357730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5278.177544] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5280.283182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5280.290587] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5280.319285] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5282.259181] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5285.352936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.357483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5285.357745] Lustre: Skipped 1 previous similar message [ 5289.671560] Lustre: server umount lustre-OST0000 complete [ 5290.465953] LustreError: 185002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.472677] LustreError: 185002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5292.043984] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5292.147825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5292.668435] LustreError: 185002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5293.978952] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5294.077550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5297.788634] LustreError: 183630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5297.797315] LustreError: 183630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5305.828535] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5305.829885] LustreError: 183631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5305.830149] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5305.831200] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5305.831207] Lustre: Skipped 2 previous similar messages [ 5305.831734] LustreError: Skipped 1 previous similar message [ 5305.838451] LustreError: 183631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5310.566615] Lustre: server umount lustre-OST0001 complete [ 5314.016953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5314.019936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5314.024332] Lustre: Skipped 1 previous similar message [ 5314.026311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5314.028412] Lustre: Skipped 1 previous similar message [ 5318.308210] Lustre: server umount lustre-MDT0000 complete [ 5319.727431] LustreError: 182016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857029 with bad export cookie 3511212596225208341 [ 5319.731062] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5319.731453] LustreError: 182016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5319.849064] Lustre: server umount lustre-MDT0001 complete [ 5328.128829] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:30:38 (1780857038) [ 5330.282665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5330.370534] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5330.383195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5330.451086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5330.453222] Lustre: Skipped 1 previous similar message [ 5331.659147] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5334.409714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5334.440987] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5334.453699] Lustre: 187799:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5335.855430] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5337.701718] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5339.076380] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5341.724641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5341.809295] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5343.990825] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5346.114759] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5348.867718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5348.896940] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5349.764580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5349.781279] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5351.253454] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5353.457990] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5354.979123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5354.980022] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5354.982297] LustreError: Skipped 1 previous similar message [ 5354.983383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5354.988112] Lustre: Skipped 2 previous similar messages [ 5354.998649] Lustre: Skipped 3 previous similar messages [ 5356.615887] Lustre: server umount lustre-OST0000 complete [ 5358.724097] LustreError: 189421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5370.339662] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5370.342786] LustreError: Skipped 1 previous similar message [ 5373.587344] Lustre: server umount lustre-OST0001 complete [ 5375.457665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.462117] Lustre: Skipped 2 previous similar messages [ 5389.280180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5389.347233] Lustre: server umount lustre-MDT0000 complete [ 5390.818385] LustreError: 187805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5390.825748] LustreError: 187805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5390.922816] LustreError: 187790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857100 with bad export cookie 3511212596225209727 [ 5390.925355] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5390.928460] LustreError: 187790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5391.084911] Lustre: server umount lustre-MDT0001 complete [ 5399.797269] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:31:50 (1780857110) [ 5405.922258] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 5409.429809] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5423.034493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5423.128236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5423.141412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5423.177957] Lustre: lustre-MDT0000: new disk, initializing [ 5423.208701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.212315] Lustre: Skipped 3 previous similar messages [ 5423.221056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5424.580766] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5428.788055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5428.831779] Lustre: 195370:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5428.856833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5428.859411] Lustre: Skipped 1 previous similar message [ 5428.896491] Lustre: lustre-MDT0001: new disk, initializing [ 5428.941939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5428.945588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5430.379579] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5433.670144] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5434.938360] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5437.482821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5437.581406] Lustre: lustre-OST0000: new disk, initializing [ 5437.583420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5437.586237] Lustre: 197112:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5439.592853] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5439.679719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5439.683462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5439.732923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5442.929077] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5444.405741] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5444.494614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5445.964320] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5446.053250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5450.210521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5450.211293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.212889] LustreError: Skipped 2 previous similar messages [ 5450.216263] Lustre: Skipped 1 previous similar message [ 5450.218322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5450.220840] Lustre: Skipped 9 previous similar messages [ 5452.885948] Lustre: server umount lustre-OST0000 complete [ 5460.379679] Lustre: server umount lustre-MDT0000 complete [ 5460.450300] LustreError: 195382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5460.457681] LustreError: 195382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5461.809260] LustreError: 196148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857171 with bad export cookie 3511212596225211400 [ 5461.811586] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5461.812726] LustreError: 196148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5461.940748] Lustre: server umount lustre-MDT0001 complete [ 5465.078613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5466.477595] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5469.407794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5470.940338] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5472.868409] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5474.082837] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5476.370729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5478.487554] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5480.550073] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5482.021625] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5491.267485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 5492.648688] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5492.724728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5495.265348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5495.271746] LustreError: Skipped 1 previous similar message [ 5499.543629] Lustre: server umount lustre-OST0000 complete [ 5501.572378] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5503.000718] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5526.883167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5527.030280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5527.037483] Lustre: Skipped 5 previous similar messages [ 5529.109204] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5530.993350] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5531.603064] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5534.674792] Lustre: server umount lustre-OST0000 complete [ 5536.225122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5536.228830] Lustre: Skipped 4 previous similar messages [ 5536.230641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5536.232549] Lustre: Skipped 6 previous similar messages [ 5538.071272] Lustre: server umount lustre-MDT0000 complete [ 5539.321331] LustreError: 199160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857249 with bad export cookie 3511212596225212268 [ 5539.324667] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5539.325972] LustreError: 199160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5539.460318] Lustre: server umount lustre-MDT0001 complete [ 5542.474683] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:34:13 (1780857253) [ 5544.911965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5546.489912] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5549.640046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5551.289936] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5553.204418] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5554.428207] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5556.725577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5558.698506] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5560.532405] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5561.878878] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5572.160826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 5573.542713] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5573.618774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5575.992854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5576.059919] Lustre: lustre-OST0001: new disk, initializing [ 5576.061503] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5576.064653] Lustre: 207739:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5578.006721] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5581.261357] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5582.740633] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5583.870922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5583.874628] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5583.902211] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5584.856720] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5586.232783] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5586.308467] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5588.964935] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5588.967666] LustreError: Skipped 2 previous similar messages [ 5593.089513] Lustre: server umount lustre-OST0001 complete [ 5594.086244] LustreError: 206270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5594.093422] LustreError: 206270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5595.147413] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5595.228840] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5596.746922] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5596.841382] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5620.407805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5622.269559] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5624.015229] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5624.543165] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5631.228861] Lustre: server umount lustre-OST0001 complete [ 5649.494384] Lustre: server umount lustre-OST0000 complete [ 5665.248132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5665.249755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5665.253290] Lustre: Skipped 19 previous similar messages [ 5665.304380] Lustre: server umount lustre-MDT0000 complete [ 5666.657397] LustreError: 204655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857376 with bad export cookie 3511212596225213185 [ 5666.660902] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5666.663540] LustreError: 204655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5672.820867] Lustre: server umount lustre-MDT0001 complete [ 5680.995326] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:36:31 (1780857391) [ 5683.303065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5683.406711] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5683.417525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5683.502917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5683.505563] Lustre: Skipped 5 previous similar messages [ 5684.860129] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5687.881602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5687.916419] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5687.926935] Lustre: 212692:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5689.411368] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5691.511658] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5692.902793] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5695.703995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5695.787768] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5698.104496] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5700.022285] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5714.285989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5714.315177] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5716.596746] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5718.646878] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5719.275441] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5721.023212] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5722.129301] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5723.578258] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5723.661037] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5724.964917] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff940e83613800.ost_server_uuid 50 [ 5725.506128] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff940e83613800.ost_server_uuid in FULL state after 0 sec [ 5725.924283] Lustre: Permanently deactivating lustre-OST0001 [ 5725.931744] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5726.820479] Lustre: Permanently reactivating lustre-OST0001 [ 5731.811221] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5731.811690] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5731.814917] Lustre: Skipped 8 previous similar messages [ 5731.817046] LustreError: Skipped 1 previous similar message [ 5733.950324] Lustre: server umount lustre-OST0001 complete [ 5741.340460] Lustre: server umount lustre-OST0000 complete [ 5748.902247] Lustre: server umount lustre-MDT0000 complete [ 5750.230247] LustreError: 213413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857460 with bad export cookie 3511212596225214445 [ 5750.233691] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5750.234488] LustreError: 213413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5750.357501] Lustre: server umount lustre-MDT0001 complete [ 5759.001740] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:37:49 (1780857469) [ 5761.806781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5761.911777] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5761.924688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5761.928657] Lustre: Skipped 5 previous similar messages [ 5763.396343] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5766.731468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.773280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5766.785319] Lustre: 218818:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5768.313593] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5770.391306] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5771.955592] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5774.527572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5774.613894] Lustre: Permanently deactivating lustre-OST0000 [ 5774.618715] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5776.680439] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5777.332429] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5778.682620] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5781.332524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5783.563922] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5785.314991] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5785.596996] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5786.232617] Lustre: Permanently reactivating lustre-OST0000 [ 5794.809264] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5794.812803] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.816960] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5794.843446] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5794.851864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5801.471118] Lustre: server umount lustre-OST0001 complete [ 5817.312294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5817.370554] Lustre: server umount lustre-OST0000 complete [ 5833.184146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5833.241170] Lustre: server umount lustre-MDT0000 complete [ 5834.553282] LustreError: 218809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857544 with bad export cookie 3511212596225216083 [ 5834.556596] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5834.558820] LustreError: 218809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5834.679363] Lustre: server umount lustre-MDT0001 complete [ 5838.469244] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5839.541781] Key type lgssc unregistered [ 5839.674312] LNet: 223202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5839.676819] LNetError: 223202:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5839.686574] LNet: Removed LNI 192.168.202.104@tcp [ 5840.022390] Key type .llcrypt unregistered [ 5840.023947] Key type ._llcrypt unregistered [ 5849.373575] Key type ._llcrypt registered [ 5849.374841] Key type .llcrypt registered [ 5849.418743] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5849.747122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5849.781021] alg: No test for adler32 (adler32-zlib) [ 5850.647457] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 5850.738875] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 5852.320170] Key type lgssc registered [ 5852.721371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5856.073748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5856.079446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.243090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5858.501235] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5861.350430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5861.398570] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5861.412408] Lustre: 224619:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5861.418488] Lustre: Permanently deactivating lustre-MDT0001 [ 5861.424266] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5861.539854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5862.893872] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5863.600824] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5863.604110] LustreError: 224659:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5863.609557] LustreError: 224659:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5863.613709] LustreError: 224659:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 5864.748747] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5866.104811] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5868.638962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5868.770242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.808509] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5871.844037] LustreError: 226243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.852607] LustreError: 226243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5872.896185] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5875.603243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5875.702524] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5877.975653] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5880.228653] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5880.808787] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5882.857498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5882.857856] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5886.020972] Lustre: Permanently reactivating lustre-MDT0001 [ 5886.027357] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5886.030827] Lustre: Skipped 2 previous similar messages [ 5887.361746] Lustre: lustre-MDT0001: Client 0a0589c4-b54a-4c8e-a518-33f71b8634b1 (at 192.168.202.4@tcp) reconnecting [ 5889.354269] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5890.007593] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5891.954474] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5892.032707] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5893.460365] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5893.535842] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5894.585809] Lustre: Permanently deactivating lustre-MDT0001 [ 5894.593160] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5894.595777] Lustre: Skipped 2 previous similar messages [ 5897.760973] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5900.258167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.261351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5900.263343] Lustre: Skipped 3 previous similar messages [ 5900.265589] Lustre: Skipped 2 previous similar messages [ 5905.378398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5905.381252] Lustre: Skipped 4 previous similar messages [ 5905.694510] Lustre: server umount lustre-MDT0000 complete [ 5907.032154] LustreError: 227719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857616 with bad export cookie 9907074615976470769 [ 5907.035425] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5907.036069] LustreError: 227719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5907.153740] Lustre: server umount lustre-MDT0001 complete [ 5914.624629] Lustre: server umount lustre-OST0000 complete [ 5918.291922] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:40:29 (1780857629) [ 5921.613293] Lustre: server umount lustre-OST0001 complete [ 5923.619270] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 5926.494243] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5938.776758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5938.868593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5938.872400] Lustre: Skipped 2 previous similar messages [ 5938.882123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5938.917414] Lustre: lustre-MDT0000: new disk, initializing [ 5938.939866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5938.946978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5940.241411] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5944.468193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.508737] Lustre: 232252:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5944.530485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5944.532764] Lustre: Skipped 1 previous similar message [ 5944.566556] Lustre: lustre-MDT0001: new disk, initializing [ 5944.609559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5944.613106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5945.919839] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5949.032935] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5950.302596] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5952.802414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5952.907289] Lustre: lustre-OST0000: new disk, initializing [ 5952.909102] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5952.912531] Lustre: 233994:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5952.942156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.945551] Lustre: Skipped 1 previous similar message [ 5955.057283] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5956.727521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5956.731210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5956.761909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5958.298515] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5958.978455] LustreError: 232259:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5963.224639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5963.291971] Lustre: lustre-OST0001: new disk, initializing [ 5963.293989] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5963.296780] Lustre: 234995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5965.502018] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5966.018462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5966.022254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5966.062787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5968.868674] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5969.072115] LustreError: 232259:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5969.080934] LustreError: 235003:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5969.441633] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5979.168117] LustreError: 235003:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5981.666695] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5981.669196] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5981.670941] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5981.673552] Lustre: Skipped 1 previous similar message [ 5985.916236] Lustre: server umount lustre-OST0001 complete [ 5986.787851] LustreError: 233978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5986.794497] LustreError: 233978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5991.907970] LustreError: 233979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5991.913825] LustreError: 233979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5997.028420] LustreError: 233978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5997.033834] LustreError: 233978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5998.049114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5998.049147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.051337] LustreError: Skipped 2 previous similar messages [ 5998.055913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5998.059109] Lustre: Skipped 3 previous similar messages [ 6002.146734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6002.147803] LustreError: 235861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6002.149323] Lustre: Skipped 1 previous similar message [ 6002.154399] LustreError: 235861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6012.384138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6012.388538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6012.388885] LustreError: 234442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6012.388893] LustreError: 234442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6012.402452] Lustre: Skipped 3 previous similar messages [ 6012.458080] Lustre: server umount lustre-OST0000 complete [ 6016.480816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6016.484378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6016.489655] Lustre: Skipped 1 previous similar message [ 6028.256081] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6028.323438] Lustre: server umount lustre-MDT0000 complete [ 6029.530362] LustreError: 232246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857739 with bad export cookie 9907074615976473359 [ 6029.532966] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6029.535325] LustreError: 232246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6035.718743] Lustre: server umount lustre-MDT0001 complete [ 6039.210342] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 6040.231440] Key type lgssc unregistered [ 6040.358359] LNet: 236899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6040.360674] LNetError: 236899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6040.367333] LNet: Removed LNI 192.168.202.104@tcp [ 6040.642081] Key type .llcrypt unregistered [ 6040.643581] Key type ._llcrypt unregistered [ 6054.727520] Key type ._llcrypt registered [ 6054.728522] Key type .llcrypt registered [ 6054.767247] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 6055.110249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6055.137061] alg: No test for adler32 (adler32-zlib) [ 6055.989422] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 6056.075415] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 6057.656145] Key type lgssc registered [ 6058.019156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6060.639699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6060.646243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.744874] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6061.751989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6061.818474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6063.128084] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6066.188746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6066.228539] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6066.242353] Lustre: 239066:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6066.351167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6067.772393] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6069.568084] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6070.786523] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6073.041024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6073.109144] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6073.169860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6075.063505] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6076.801757] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6079.499113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6125.539102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6125.540903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6125.541729] LustreError: Skipped 1 previous similar message [ 6125.542389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6125.546371] Lustre: Skipped 1 previous similar message [ 6127.456770] Lustre: server umount lustre-OST0000 complete [ 6128.886282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6132.736254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6132.872117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6134.709990] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6136.379854] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6145.124120] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.128466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.128621] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6145.134590] Lustre: Skipped 1 previous similar message [ 6171.105542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6171.106769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.108356] LustreError: Skipped 1 previous similar message [ 6171.115230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6171.118120] Lustre: Skipped 2 previous similar messages [ 6175.515675] Lustre: server umount lustre-OST0000 complete [ 6179.296931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6179.299300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6179.302691] Lustre: Skipped 1 previous similar message [ 6179.304304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6181.345139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6181.345497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6186.465019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6186.467397] Lustre: Skipped 2 previous similar messages [ 6191.072112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6191.126312] Lustre: server umount lustre-MDT0000 complete [ 6192.316460] LustreError: 239788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857902 with bad export cookie 13708093992903218095 [ 6192.318518] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6192.320249] LustreError: 239788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6192.429989] Lustre: server umount lustre-MDT0001 complete [ 6195.987108] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 6196.917513] Key type lgssc unregistered [ 6197.036348] LNet: 244020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6197.038361] LNetError: 244020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6197.044246] LNet: Removed LNI 192.168.202.104@tcp [ 6197.328892] Key type .llcrypt unregistered [ 6197.330371] Key type ._llcrypt unregistered [ 6206.600628] Key type ._llcrypt registered [ 6206.602299] Key type .llcrypt registered [ 6206.654304] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 6206.993398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6207.017067] alg: No test for adler32 (adler32-zlib) [ 6207.871869] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 6207.951778] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 6209.528169] Key type lgssc registered [ 6209.950705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6212.978046] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6212.992078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.187935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6215.503856] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6218.466031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6218.589183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6219.924598] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6221.585068] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6222.752089] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6225.002971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6225.135252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6227.103345] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6228.980765] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6239.461404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6254.048868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6254.051802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.055877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6255.074641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.075928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6255.080527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6255.081707] Lustre: Skipped 1 previous similar message [ 6258.003581] Lustre: server umount lustre-OST0000 complete [ 6259.680829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6259.684202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6259.687846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6265.313084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6265.315046] Lustre: Skipped 3 previous similar messages [ 6265.362072] Lustre: server umount lustre-MDT0000 complete [ 6266.520214] LustreError: 245422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857976 with bad export cookie 1363988282635873465 [ 6266.523026] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6266.525817] LustreError: 245422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6266.632703] Lustre: server umount lustre-MDT0001 complete [ 6269.965224] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 6270.918436] Key type lgssc unregistered [ 6271.038378] LNet: 249365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6271.041272] LNetError: 249365:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6271.050269] LNet: Removed LNI 192.168.202.104@tcp [ 6271.338276] Key type .llcrypt unregistered [ 6271.339348] Key type ._llcrypt unregistered [ 6278.117329] Key type ._llcrypt registered [ 6278.118519] Key type .llcrypt registered [ 6278.153123] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 6278.513589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6278.553777] alg: No test for adler32 (adler32-zlib) [ 6279.406480] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 6279.484960] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 6281.064146] Key type lgssc registered [ 6281.404849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6283.879757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6283.885662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6285.033274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6286.325681] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6289.009783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6289.120111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6290.428922] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6292.158641] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6293.350414] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6295.594540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6295.700664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6297.548711] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6299.262933] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6310.056233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6320.611663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6320.612645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.614530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6320.615325] LustreError: Skipped 1 previous similar message [ 6320.619618] Lustre: Skipped 1 previous similar message [ 6324.946337] Lustre: server umount lustre-OST0000 complete [ 6330.336833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6330.340423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6330.346924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6330.348743] Lustre: Skipped 1 previous similar message [ 6332.306277] Lustre: server umount lustre-MDT0000 complete [ 6333.502068] LustreError: 250527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858043 with bad export cookie 7516849062277124315 [ 6333.504879] LustreError: 250527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6333.505021] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.612287] Lustre: server umount lustre-MDT0001 complete [ 6337.203916] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 6338.234344] Key type lgssc unregistered [ 6338.376372] LNet: 253886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6338.378821] LNetError: 253886:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6338.386317] LNet: Removed LNI 192.168.202.104@tcp [ 6338.703852] Key type .llcrypt unregistered [ 6338.705085] Key type ._llcrypt unregistered [ 6347.824860] Key type ._llcrypt registered [ 6347.826519] Key type .llcrypt registered [ 6347.870194] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 6348.269951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6348.316122] alg: No test for adler32 (adler32-zlib) [ 6349.177299] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 6349.275545] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 6350.856169] Key type lgssc registered [ 6351.247246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6353.940906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6353.947558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6355.135306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6356.526113] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6359.636476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6359.768433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6361.137632] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6363.101934] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6364.412072] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6366.808619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6366.926918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6368.916496] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6370.809603] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6381.290518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6395.873396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6395.876025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6395.880429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6396.897769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6396.898410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6396.899802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6396.906433] Lustre: Skipped 1 previous similar message [ 6400.277882] Lustre: server umount lustre-OST0000 complete [ 6402.017693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.018642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6402.023249] Lustre: Skipped 1 previous similar message [ 6407.137220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6407.139632] Lustre: Skipped 2 previous similar messages [ 6407.769742] Lustre: server umount lustre-MDT0000 complete [ 6409.143222] LustreError: 256874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858119 with bad export cookie 8169433257585439192 [ 6409.146613] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6409.147616] LustreError: 256874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6409.265186] Lustre: server umount lustre-MDT0001 complete [ 6413.202927] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 6414.346621] Key type lgssc unregistered [ 6414.472378] LNet: 259230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6414.475388] LNetError: 259230:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6414.483268] LNet: Removed LNI 192.168.202.104@tcp [ 6414.812698] Key type .llcrypt unregistered [ 6414.814135] Key type ._llcrypt unregistered [ 6422.382041] Key type ._llcrypt registered [ 6422.383039] Key type .llcrypt registered [ 6422.423714] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 6422.819241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6422.856441] alg: No test for adler32 (adler32-zlib) [ 6423.717942] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 6423.802195] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 6425.384185] Key type lgssc registered [ 6425.784309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6428.675572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6428.680875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.848399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6431.171824] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6434.322253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.459807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6435.858864] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6437.900212] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6439.161141] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6441.433307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6441.536675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6443.466896] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6445.367489] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6454.888909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6460.389873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6460.391939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6460.392976] LustreError: Skipped 1 previous similar message [ 6460.397602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6460.401221] Lustre: Skipped 1 previous similar message [ 6461.715330] Lustre: server umount lustre-OST0000 complete [ 6465.116356] Lustre: server umount lustre-MDT0000 complete [ 6465.505780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6465.505865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6465.506641] LustreError: 260423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6465.506650] LustreError: 260423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6465.520820] Lustre: Skipped 2 previous similar messages [ 6466.353195] LustreError: 260395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858176 with bad export cookie 5966821757976379768 [ 6466.356793] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6466.358059] LustreError: 260395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6466.484489] Lustre: server umount lustre-MDT0001 complete [ 6470.421740] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 6471.525498] Key type lgssc unregistered [ 6471.656496] LNet: 263788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6471.659785] LNetError: 263788:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6471.668388] LNet: Removed LNI 192.168.202.104@tcp [ 6471.998620] Key type .llcrypt unregistered [ 6471.999752] Key type ._llcrypt unregistered [ 6475.947464] Key type ._llcrypt registered [ 6475.948513] Key type .llcrypt registered [ 6475.990365] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6481.385133] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 6487.457928] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 6487.811829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6487.825152] alg: No test for adler32 (adler32-zlib) [ 6488.694076] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 6488.776906] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 6490.368207] Key type lgssc registered [ 6490.725848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6501.870912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6501.877691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6502.964264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6502.975095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6503.007068] Lustre: lustre-MDT0000: new disk, initializing [ 6503.028981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6503.035773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6504.217531] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6508.159024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.194214] Lustre: 267262:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6508.217593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6508.219563] Lustre: Skipped 1 previous similar message [ 6508.253685] Lustre: lustre-MDT0001: new disk, initializing [ 6508.284703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6508.296788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6508.300163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6509.558426] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6512.580243] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6513.816880] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6515.992350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6516.088473] Lustre: lustre-OST0000: new disk, initializing [ 6516.090704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6516.092813] Lustre: 269002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6516.117254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6516.978190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6516.980888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6517.013704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6517.790316] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6520.856585] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6522.239169] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6522.325083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6523.654645] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6523.728394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6527.458741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6527.460803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6527.463177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6527.463756] Lustre: Skipped 1 previous similar message [ 6530.514489] Lustre: server umount lustre-OST0000 complete [ 6532.577670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6532.578602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6532.580988] Lustre: Skipped 1 previous similar message [ 6532.585663] Lustre: Skipped 2 previous similar messages [ 6537.697193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6537.699141] Lustre: Skipped 1 previous similar message [ 6537.879501] Lustre: server umount lustre-MDT0000 complete [ 6539.227400] LustreError: 269001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858249 with bad export cookie 17156819667331410694 [ 6539.230959] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6539.231349] LustreError: 269001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6539.347805] Lustre: server umount lustre-MDT0001 complete [ 6542.263356] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:50:53 (1780858253) [ 6544.387758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6544.554307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6545.904783] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6548.663562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6548.779956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6550.147176] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6551.891275] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6553.044583] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6555.250492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6557.115016] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6558.741624] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6579.169042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6579.172212] LustreError: Skipped 1 previous similar message [ 6579.174239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.179514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6581.217877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6581.220079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6583.120915] Lustre: server umount lustre-OST0000 complete [ 6584.801555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6584.804477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6584.806938] Lustre: Skipped 2 previous similar messages [ 6586.339217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6586.343540] Lustre: Skipped 1 previous similar message [ 6590.620176] Lustre: server umount lustre-MDT0000 complete [ 6591.456713] LustreError: 271263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6591.461384] LustreError: 271263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6591.822187] LustreError: 271244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858301 with bad export cookie 17156819667331411562 [ 6591.825080] LustreError: 271244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6591.825508] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6591.825576] LustreError: 271982:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 6591.825727] LustreError: 271982:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000075523d0c x1867365165887232/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6591.939218] Lustre: server umount lustre-MDT0001 complete [ 6595.368349] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 6596.395538] Key type lgssc unregistered [ 6596.525316] LNet: 274458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6596.527951] LNetError: 274458:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6596.534308] LNet: Removed LNI 192.168.202.104@tcp [ 6596.816864] Key type .llcrypt unregistered [ 6596.818081] Key type ._llcrypt unregistered [ 6605.036556] Key type ._llcrypt registered [ 6605.037914] Key type .llcrypt registered [ 6605.075449] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 6610.957852] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 6611.358462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6611.364675] alg: No test for adler32 (adler32-zlib) [ 6612.213076] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 6612.290662] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 6613.864146] Key type lgssc registered [ 6614.205410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6628.368576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6628.374519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6629.468197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6629.480258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6629.513306] Lustre: lustre-MDT0000: new disk, initializing [ 6629.536376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6629.544441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6630.876689] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6635.022534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6635.062662] Lustre: 278218:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6635.087287] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6635.089293] Lustre: Skipped 1 previous similar message [ 6635.125758] Lustre: lustre-MDT0001: new disk, initializing [ 6635.159441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6635.171240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6635.174668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6636.438570] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6639.552318] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6640.696486] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6642.986408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6643.082675] Lustre: lustre-OST2710: new disk, initializing [ 6643.084432] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6643.087286] Lustre: 279957:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6643.118970] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6644.279508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6644.282776] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6644.308503] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6644.931601] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6647.987274] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6650.301500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6650.367935] Lustre: lustre-OST03e8: new disk, initializing [ 6650.370463] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6650.373964] Lustre: 280910:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6650.406918] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6652.272769] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6652.359406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6652.362929] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6652.401174] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6654.074885] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6656.609704] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6656.688939] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6657.894371] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6657.959433] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6659.171829] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6659.236531] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6660.464674] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6660.523056] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6662.627129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6662.628605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6662.630816] Lustre: Skipped 3 previous similar messages [ 6662.633195] Lustre: Skipped 2 previous similar messages [ 6667.611493] Lustre: server umount lustre-MDT0000 complete [ 6668.780991] LustreError: 278212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858378 with bad export cookie 11098802081522871529 [ 6668.784711] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6668.784924] LustreError: 278212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6668.884260] Lustre: server umount lustre-MDT0001 complete [ 6681.090361] Lustre: server umount lustre-OST2710 complete [ 6692.706733] Lustre: server umount lustre-OST03e8 complete [ 6698.616651] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 6701.897639] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 6713.408401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6713.491871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6713.502913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6713.536812] Lustre: lustre-MDT0000: new disk, initializing [ 6713.559086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6713.566175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6714.791089] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6718.838949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6718.879599] Lustre: 285935:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6718.994522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6720.231328] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6723.205991] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6724.331750] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6726.538829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6726.649829] Lustre: lustre-OST0000: new disk, initializing [ 6726.651037] Lustre: Skipped 1 previous similar message [ 6726.652268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6726.653914] Lustre: Skipped 2 previous similar messages [ 6726.655842] Lustre: 287673:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6726.685384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.687608] Lustre: Skipped 1 previous similar message [ 6728.440061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6728.442588] Lustre: Skipped 1 previous similar message [ 6728.444288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6728.485148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6728.612185] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6731.720496] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6733.102452] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6733.174221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6734.463488] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6734.535292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6738.914054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6738.915402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6738.916610] LustreError: Skipped 1 previous similar message [ 6738.917263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6738.920455] Lustre: Skipped 1 previous similar message [ 6741.272358] Lustre: server umount lustre-OST0000 complete [ 6744.033459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6744.034139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6744.038223] Lustre: Skipped 1 previous similar message [ 6744.041754] Lustre: Skipped 2 previous similar messages [ 6748.635965] Lustre: server umount lustre-MDT0000 complete [ 6749.152920] LustreError: 285944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6749.158683] LustreError: 285944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6749.872688] LustreError: 285925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858459 with bad export cookie 11098802081522873195 [ 6749.874719] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6749.876588] LustreError: 285925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6749.998709] Lustre: server umount lustre-MDT0001 complete [ 6752.726381] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 14:54:23 (1780858463) [ 6753.213163] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6753.742534] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 14:54:24 (1780858464) [ 6754.746628] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 6763.244058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6763.325694] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6763.332388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6763.387073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6764.667570] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6767.345385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.378557] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6767.387217] Lustre: 291363:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6768.672826] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6784.355089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6784.424583] Lustre: Denying initial registration attempt from nid 192.168.202.104@tcp, specified as failover [ 6784.428647] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6784.432381] LustreError: 292780:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 6784.435119] LustreError: 292780:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 6784.437417] LustreError: 292780:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6784.459688] Lustre: server umount lustre-OST0000 complete [ 6784.461130] LustreError: 292780:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6788.064684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6788.067022] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.070664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6792.786860] Lustre: server umount lustre-MDT0000 complete [ 6793.936299] LustreError: 291356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858503 with bad export cookie 11098802081522874049 [ 6793.938983] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6793.940773] LustreError: 291356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6794.045916] Lustre: server umount lustre-MDT0001 complete [ 6797.490732] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 6798.429113] Key type lgssc unregistered [ 6798.559616] LNet: 293882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6798.563262] LNetError: 293882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6798.573363] LNet: Removed LNI 192.168.202.104@tcp [ 6798.854218] Key type .llcrypt unregistered [ 6798.855585] Key type ._llcrypt unregistered [ 6802.190764] Key type ._llcrypt registered [ 6802.192030] Key type .llcrypt registered [ 6802.229908] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 6802.630573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6802.665311] alg: No test for adler32 (adler32-zlib) [ 6803.519397] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 6803.590520] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 6805.168140] Key type lgssc registered [ 6805.502325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6828.855922] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 6831.578636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6831.583845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6832.682150] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6832.693644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6832.759835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6834.049879] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6836.757242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.787360] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6836.797877] Lustre: 296592:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6836.884996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6838.073909] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6839.731430] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6840.879172] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6856.098921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6856.164511] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6856.227724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6858.017199] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6859.743496] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6860.464473] Lustre: server umount lustre-OST0000 complete [ 6862.816802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6862.818941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6862.823250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6866.913509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6866.913631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6866.920069] Lustre: Skipped 1 previous similar message [ 6867.795092] Lustre: server umount lustre-MDT0000 complete [ 6869.008334] LustreError: 296582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858578 with bad export cookie 5018395031410312603 [ 6869.011603] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6869.012465] LustreError: 296582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6869.128271] Lustre: server umount lustre-MDT0001 complete [ 6872.794574] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 6873.810169] Key type lgssc unregistered [ 6873.935477] LNet: 299797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6873.938260] LNetError: 299797:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6873.950394] LNet: Removed LNI 192.168.202.104@tcp [ 6874.257728] Key type .llcrypt unregistered [ 6874.258724] Key type ._llcrypt unregistered [ 6883.533275] Key type ._llcrypt registered [ 6883.534530] Key type .llcrypt registered [ 6883.574609] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 6883.927228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6883.963102] alg: No test for adler32 (adler32-zlib) [ 6884.825001] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 6884.908331] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 6886.488171] Key type lgssc registered [ 6886.856781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6889.364894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6889.370277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6890.532927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6891.819087] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6894.696402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.831249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6896.107359] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6897.899452] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6899.208243] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6903.651177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6903.774406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6905.656431] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6907.582573] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6924.417293] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 6925.792884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6925.795939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6925.799878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6926.306793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6926.311311] Lustre: Skipped 1 previous similar message [ 6927.445382] Lustre: server umount lustre-MDT0000 complete [ 6928.829230] LustreError: 302788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858638 with bad export cookie 9906426959500901634 [ 6928.831704] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6928.833262] LustreError: 302788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6929.530454] LustreError: 301167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.537269] LustreError: 301167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6929.538894] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.4@tcp (stopping) [ 6929.543912] Lustre: Skipped 3 previous similar messages [ 6931.425026] LustreError: 303321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6931.425247] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6933.472706] LustreError: 301168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6933.472996] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6933.481423] Lustre: Skipped 1 previous similar message [ 6938.593337] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6938.593487] LustreError: 302785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6938.595960] Lustre: Skipped 1 previous similar message [ 6938.602546] LustreError: 302785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6943.200223] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6943.279605] Lustre: server umount lustre-MDT0001 complete [ 6945.462441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6948.283713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6953.952841] LustreError: 300483:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000038eda2d7 x1867365581089024/t0(0) o250->MGC192.168.202.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6954.086921] LustreError: 302775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6954.093614] LustreError: 302775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6954.110838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6954.134053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6955.386390] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6958.259172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6958.379251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6959.675677] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 6961.448113] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6968.806780] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6971.170902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 6972.341267] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6972.887052] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6974.944859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6974.948364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.952454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6974.954252] Lustre: Skipped 1 previous similar message [ 6979.041920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6979.044456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6980.178935] Lustre: server umount lustre-OST0000 complete [ 6984.164584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6987.608839] Lustre: server umount lustre-MDT0000 complete [ 6988.828148] LustreError: 302788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858698 with bad export cookie 9906426959500908753 [ 6988.831876] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6988.832293] LustreError: 302788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6988.928674] Lustre: server umount lustre-MDT0001 complete [ 6992.361028] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 6993.384670] Key type lgssc unregistered [ 6993.497402] LNet: 306872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6993.499934] LNetError: 306872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6993.508367] LNet: Removed LNI 192.168.202.104@tcp [ 6993.805062] Key type .llcrypt unregistered [ 6993.806221] Key type ._llcrypt unregistered [ 7002.679722] Key type ._llcrypt registered [ 7002.680629] Key type .llcrypt registered [ 7002.717668] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 7003.211991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7003.223358] alg: No test for adler32 (adler32-zlib) [ 7004.069228] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 7004.152427] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 7005.728192] Key type lgssc registered [ 7006.087699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7008.582946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7008.588964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7009.738829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7010.957322] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7013.668447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7013.783856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7015.084043] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7016.859516] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7017.979566] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7018.936106] LustreError: 308277:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7018.939117] LustreError: 308277:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7018.942167] LustreError: 308277:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 7018.977329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7018.977623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7018.978795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7018.985621] Lustre: Skipped 1 previous similar message [ 7023.073230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7023.075643] Lustre: Skipped 2 previous similar messages [ 7028.193132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7028.195640] Lustre: Skipped 1 previous similar message [ 7033.312122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7033.312910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7033.317259] Lustre: Skipped 1 previous similar message [ 7033.366737] Lustre: server umount lustre-MDT0000 complete [ 7034.553499] LustreError: 308228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858744 with bad export cookie 11596880190129727638 [ 7034.556499] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7034.556981] LustreError: 308228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7034.668636] Lustre: server umount lustre-MDT0001 complete [ 7039.539272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7039.631924] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7039.644288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7039.718126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7040.980079] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7043.807907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7043.841659] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7043.851679] Lustre: 310606:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7043.945253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7045.193141] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7046.961308] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7048.134065] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7052.770579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7052.839115] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7052.843097] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7052.846446] LustreError: 312263:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 7052.848932] LustreError: 312263:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7052.851739] LustreError: 312263:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7052.877935] Lustre: server umount lustre-OST0000 complete [ 7052.879497] LustreError: 312263:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7057.827240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7057.891645] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7057.955543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7059.839419] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7061.615253] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7064.265149] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7064.299056] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7064.328444] Lustre: lustre-OST0001: new disk, initializing [ 7064.330227] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7064.333159] Lustre: 313678:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7066.005982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7066.011371] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7066.043082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 7066.070659] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7066.365750] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7068.185779] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7071.202635] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7071.205560] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7071.209837] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7071.212073] Lustre: Skipped 1 previous similar message [ 7075.327937] Lustre: server umount lustre-OST0001 complete [ 7076.325281] LustreError: 312728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7076.330099] LustreError: 312728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7081.442776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7081.443746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7081.443747] LustreError: 313959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7081.443761] LustreError: 313959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7081.447237] Lustre: Skipped 1 previous similar message [ 7081.449845] LustreError: Skipped 2 previous similar messages [ 7081.450496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7081.462099] Lustre: Skipped 1 previous similar message [ 7082.700014] Lustre: server umount lustre-OST0000 complete [ 7085.024947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7085.027682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.336110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7098.395417] Lustre: server umount lustre-MDT0000 complete [ 7099.586040] LustreError: 312744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858809 with bad export cookie 11596880190129728156 [ 7099.588574] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7099.589226] LustreError: 312744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7099.700503] Lustre: server umount lustre-MDT0001 complete [ 7103.124384] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 7104.070941] Key type lgssc unregistered [ 7104.185324] LNet: 315510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7104.188130] LNetError: 315510:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7104.195310] LNet: Removed LNI 192.168.202.104@tcp [ 7104.463797] Key type .llcrypt unregistered [ 7104.465159] Key type ._llcrypt unregistered [ 7134.901404] Key type ._llcrypt registered [ 7134.903073] Key type .llcrypt registered [ 7134.940194] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 7141.181258] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 7141.728624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7141.751185] alg: No test for adler32 (adler32-zlib) [ 7142.616890] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 7142.713849] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 7144.304198] Key type lgssc registered [ 7144.692288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7156.331992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7156.337336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7157.424862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7157.435965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7157.470911] Lustre: lustre-MDT0000: new disk, initializing [ 7157.495968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7157.503270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7158.768538] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7162.918345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7162.961445] Lustre: 320388:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7162.983611] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7162.985787] Lustre: Skipped 1 previous similar message [ 7163.021211] Lustre: lustre-MDT0001: new disk, initializing [ 7163.059170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7163.072462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7163.076544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7164.334358] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7167.330986] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7168.501351] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7170.676737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7170.774219] Lustre: lustre-OST0000: new disk, initializing [ 7170.775795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7170.778604] Lustre: 322131:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7170.806285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7172.630074] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7174.709997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7174.713143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7174.744622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7175.705912] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7176.947914] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7177.017405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7178.267167] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7178.336553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7179.746227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7179.747029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7179.748536] LustreError: Skipped 1 previous similar message [ 7179.752557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7179.755150] Lustre: Skipped 1 previous similar message [ 7181.070968] Lustre: server umount lustre-OST0000 complete [ 7183.840782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7183.843312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7183.846410] Lustre: Skipped 1 previous similar message [ 7183.848126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7184.865217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7184.865862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7188.379789] Lustre: server umount lustre-MDT0000 complete [ 7189.631113] LustreError: 322130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858899 with bad export cookie 3103282884347135853 [ 7189.632797] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7189.635202] LustreError: 322130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7189.750613] Lustre: server umount lustre-MDT0001 complete [ 7192.479225] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:01:43 (1780858903) [ 7194.940937] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:01:45 (1780858905) [ 7197.851645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7198.028119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7199.339615] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7202.113653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7202.241418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7203.484510] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7205.243546] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7206.477861] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7208.669457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7210.554174] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7212.314184] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7226.275929] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7226.838192] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7227.366587] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7227.893260] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7228.514334] Lustre: Failing over lustre-MDT0000 [ 7228.714785] Lustre: server umount lustre-MDT0000 complete [ 7232.482524] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7232.483269] LustreError: 324728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7232.487540] Lustre: Skipped 2 previous similar messages [ 7232.493848] LustreError: 324728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7233.152849] LustreError: 324728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7237.601489] LustreError: 324734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7237.605658] LustreError: 324734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7241.642191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7241.683899] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7241.771918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7241.774206] Lustre: Skipped 1 previous similar message [ 7241.799182] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7243.097971] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7243.385983] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7246.821325] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7246.832331] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7246.849522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7248.031300] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7248.581411] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7250.459296] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7251.066683] Lustre: Failing over lustre-MDT0000 [ 7251.299438] Lustre: server umount lustre-MDT0000 complete [ 7251.938683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7251.939414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7251.940189] LustreError: 324728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7251.940197] LustreError: 324728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7251.943627] Lustre: Skipped 1 previous similar message [ 7257.057147] LustreError: 324752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7257.061821] LustreError: 324752:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7263.924038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7263.956278] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7264.038234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7264.060790] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7265.231194] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7268.986335] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7269.352675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7269.355192] Lustre: Skipped 2 previous similar messages [ 7269.363071] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7269.378458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7270.443465] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7270.962979] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7274.466404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7274.467315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.468077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7274.468081] Lustre: Skipped 1 previous similar message [ 7274.475690] Lustre: Skipped 2 previous similar messages [ 7279.254097] Lustre: server umount lustre-MDT0000 complete [ 7279.585074] LustreError: 324733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7279.590617] LustreError: 324733:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 7280.443510] LustreError: 324715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780858990 with bad export cookie 3103282884347138534 [ 7280.446319] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7280.447511] LustreError: 324715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7280.552560] Lustre: server umount lustre-MDT0001 complete [ 7292.055792] Lustre: server umount lustre-OST0000 complete [ 7296.801236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7296.961457] LustreError: 330519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7296.985560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7298.238941] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7300.828902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7302.078436] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7303.703427] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7304.822386] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7307.004984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7309.056803] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7310.849338] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7321.446424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7330.784875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7330.787723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7330.791734] Lustre: Skipped 1 previous similar message [ 7330.793244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7330.795268] Lustre: Skipped 2 previous similar messages [ 7333.580409] Lustre: server umount lustre-OST0000 complete [ 7342.049133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7342.051479] Lustre: Skipped 5 previous similar messages [ 7349.216134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7349.278408] Lustre: server umount lustre-MDT0000 complete [ 7350.472135] LustreError: 330500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859060 with bad export cookie 3103282884347139241 [ 7350.474430] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7350.476441] LustreError: 330500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7350.575258] Lustre: server umount lustre-MDT0001 complete [ 7353.816981] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 7354.803675] Key type lgssc unregistered [ 7354.919405] LNet: 333718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7354.921906] LNetError: 333718:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7354.930300] LNet: Removed LNI 192.168.202.104@tcp [ 7355.212892] Key type .llcrypt unregistered [ 7355.214546] Key type ._llcrypt unregistered [ 7364.145298] Key type ._llcrypt registered [ 7364.146455] Key type .llcrypt registered [ 7364.182396] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 7364.586575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7364.600486] alg: No test for adler32 (adler32-zlib) [ 7365.461777] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 7365.543667] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 7367.128158] Key type lgssc registered [ 7367.467150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7369.891581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7369.897532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.039870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7372.243574] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7374.953711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7375.065679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7376.307424] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7378.135125] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7379.315811] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7381.501072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7381.606258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7383.475567] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7385.150093] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7395.940665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7406.048789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7406.050563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7406.054292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7406.561911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7406.562633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7406.565684] Lustre: Skipped 1 previous similar message [ 7410.516918] Lustre: server umount lustre-MDT0000 complete [ 7411.613453] LustreError: 335175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859121 with bad export cookie 11114197284337178357 [ 7411.616168] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7411.618925] LustreError: 335175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7411.681835] LustreError: 335190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7411.682534] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7411.686384] LustreError: 335190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7411.726731] Lustre: server umount lustre-MDT0001 complete [ 7419.028235] Lustre: server umount lustre-OST0000 complete [ 7423.225288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7423.363872] LustreError: 338584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7423.385222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7424.633536] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7427.345397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7427.456200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7428.679668] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7430.308255] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7431.455236] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7433.573991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7434.727624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7435.447200] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7437.291970] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7448.035669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7448.037058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7448.039202] Lustre: Skipped 2 previous similar messages [ 7448.040870] Lustre: Skipped 2 previous similar messages [ 7448.042601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7450.199185] Lustre: server umount lustre-MDT0000 complete [ 7451.488292] LustreError: 338564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859161 with bad export cookie 11114197284337179456 [ 7451.491717] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7451.492152] LustreError: 338564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7451.611283] Lustre: server umount lustre-MDT0001 complete [ 7463.133134] Lustre: server umount lustre-OST0000 complete [ 7468.246288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7468.400983] LustreError: 342150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7468.423582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7468.426082] Lustre: Skipped 1 previous similar message [ 7469.799476] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7472.707027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7474.183909] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7475.953649] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7477.136611] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7479.379749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7481.265328] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7482.936690] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7494.888080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7500.260309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7500.261276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7500.268437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7500.270292] Lustre: Skipped 1 previous similar message [ 7501.774485] Lustre: server umount lustre-OST0000 complete [ 7503.840962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7505.185990] Lustre: server umount lustre-MDT0000 complete [ 7505.377484] LustreError: 342150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7505.381674] LustreError: 342150:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7506.447715] LustreError: 342176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859216 with bad export cookie 11114197284337180464 [ 7506.453549] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7506.578181] Lustre: server umount lustre-MDT0001 complete [ 7510.210344] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 7511.223775] Key type lgssc unregistered [ 7511.344393] LNet: 345345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7511.347111] LNetError: 345345:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7511.353286] LNet: Removed LNI 192.168.202.104@tcp [ 7511.678472] Key type .llcrypt unregistered [ 7511.679727] Key type ._llcrypt unregistered [ 7521.677905] Key type ._llcrypt registered [ 7521.679016] Key type .llcrypt registered [ 7521.719082] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 7522.154077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7522.198540] alg: No test for adler32 (adler32-zlib) [ 7523.051186] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 7523.132331] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 7524.712132] Key type lgssc registered [ 7525.052046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7527.650123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7527.655798] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7527.658714] LustreError: 346744:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7527.667080] LustreError: 346744:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7527.669553] LustreError: 346744:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7527.672022] LustreError: 346744:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7527.674888] LustreError: 346744:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7530.212171] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7530.215855] LustreError: 347140:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7530.223844] LustreError: 347140:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7530.226514] LustreError: 347140:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7530.228671] LustreError: 347140:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7530.231795] LustreError: 347140:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7536.611344] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 7537.667873] Key type lgssc unregistered [ 7537.790429] LNet: 348124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7537.792959] LNetError: 348124:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7537.801353] LNet: Removed LNI 192.168.202.104@tcp [ 7538.112479] Key type .llcrypt unregistered [ 7538.113563] Key type ._llcrypt unregistered [ 7544.157739] Key type ._llcrypt registered [ 7544.158687] Key type .llcrypt registered [ 7544.197484] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 7550.362950] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 7550.866754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7550.898928] alg: No test for adler32 (adler32-zlib) [ 7551.761428] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 7551.846656] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 7553.432155] Key type lgssc registered [ 7553.793194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7565.071606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7565.079383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7566.165324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7566.177034] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7566.214876] Lustre: lustre-MDT0000: new disk, initializing [ 7566.242079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7566.250759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7567.571323] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7571.712302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7571.756248] Lustre: 351216:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7571.778824] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7571.780713] Lustre: Skipped 1 previous similar message [ 7571.814522] Lustre: lustre-MDT0001: new disk, initializing [ 7571.846843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7571.858690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7571.862155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7573.267388] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7576.491156] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7577.640747] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7579.922238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7580.032090] Lustre: lustre-OST0000: new disk, initializing [ 7580.034622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7580.038555] Lustre: 352959:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7580.069610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7581.818347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7581.821341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7581.858934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7581.960869] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7584.925840] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7586.263768] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7586.344074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7587.542717] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7587.609943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7592.419229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7592.419889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7592.426137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7592.428943] Lustre: Skipped 1 previous similar message [ 7594.320583] Lustre: server umount lustre-OST0000 complete [ 7597.537822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7597.537967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7597.540526] LustreError: Skipped 1 previous similar message [ 7597.544647] Lustre: Skipped 1 previous similar message [ 7597.548258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7597.549754] Lustre: Skipped 1 previous similar message [ 7601.697734] Lustre: server umount lustre-MDT0000 complete [ 7602.889282] LustreError: 351209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859312 with bad export cookie 16860461571429280438 [ 7602.892413] LustreError: 351209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7602.892855] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7603.006389] Lustre: server umount lustre-MDT0001 complete [ 7605.892342] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7606.455572] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:08:37 (1780859317) [ 7608.694121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7608.873977] LustreError: 355267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7608.882430] LustreError: 355267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7608.899599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7610.242936] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7613.046717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7613.171422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7614.486656] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7616.217711] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7617.492571] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7619.739543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7621.662722] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7623.489564] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7625.729682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7625.796052] Lustre: lustre-OST0001: new disk, initializing [ 7625.797693] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7625.800094] Lustre: 357761:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7625.828964] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7625.831165] Lustre: Skipped 1 previous similar message [ 7627.785794] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7630.839699] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7631.360461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7631.364637] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7631.390653] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7638.501033] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7638.502118] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7638.504643] Lustre: Skipped 2 previous similar messages [ 7641.218199] Lustre: lustre-OST0001: Not available for connect from 192.168.202.4@tcp (stopping) [ 7641.220100] Lustre: Skipped 1 previous similar message [ 7646.331742] Lustre: lustre-OST0001: Not available for connect from 192.168.202.4@tcp (stopping) [ 7646.335134] Lustre: Skipped 2 previous similar messages [ 7651.296088] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7651.334066] Lustre: server umount lustre-OST0001 complete [ 7651.451191] LustreError: 356860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7653.860876] LustreError: 356861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7653.866169] LustreError: 356861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7658.981670] LustreError: 356862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7658.986404] LustreError: 356862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7664.099032] LustreError: 358368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7664.099981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7664.101380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7664.101385] Lustre: Skipped 2 previous similar messages [ 7664.103480] LustreError: 358368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7664.107015] Lustre: Skipped 1 previous similar message [ 7664.107254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7669.520189] Lustre: server umount lustre-OST0000 complete [ 7674.337942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7684.577223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7684.579419] Lustre: Skipped 8 previous similar messages [ 7685.088113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7685.152115] Lustre: server umount lustre-MDT0000 complete [ 7686.404125] LustreError: 355250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859396 with bad export cookie 16860461571429281313 [ 7686.407337] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7686.408207] LustreError: 355250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7692.598730] Lustre: server umount lustre-MDT0001 complete [ 7696.285510] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 7697.307955] Key type lgssc unregistered [ 7697.433482] LNet: 359616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7697.435968] LNetError: 359616:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7697.444510] LNet: Removed LNI 192.168.202.104@tcp [ 7697.755260] Key type .llcrypt unregistered [ 7697.756362] Key type ._llcrypt unregistered [ 7715.304326] Key type ._llcrypt registered [ 7715.305401] Key type .llcrypt registered [ 7715.342458] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 7715.714814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7715.741362] alg: No test for adler32 (adler32-zlib) [ 7716.590947] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 7716.671048] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 7718.248153] Key type lgssc registered [ 7718.629656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7721.178546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7721.184433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7722.271528] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7722.279651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7722.339580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7723.550027] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7726.293935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7726.326680] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7726.338659] Lustre: 362374:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7726.437434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7727.708116] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7729.406190] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7730.662113] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7731.681103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7731.683559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7731.687669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7734.241793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7734.242423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7737.498790] Lustre: server umount lustre-MDT0000 complete [ 7738.711147] LustreError: 362364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859448 with bad export cookie 589275977384060574 [ 7738.714653] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7738.715256] LustreError: 362364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7738.822039] Lustre: server umount lustre-MDT0001 complete [ 7741.815206] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:10:52 (1780859452) [ 7743.962315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7744.139802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7745.401631] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7748.249059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7748.370787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7749.646426] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7751.328223] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7752.489818] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7754.678331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7754.745671] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7754.807181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7756.702810] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7758.566418] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7767.627330] Lustre: Permanently deactivating lustre-OST0000 [ 7767.634694] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7770.160830] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7771.006279] LustreError: 367209:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7771.014174] LustreError: 367209:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 7771.019107] LustreError: 367209:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7771.740563] LustreError: 367257:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 7771.744846] LustreError: 367257:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7778.547566] Lustre: server umount lustre-OST0000 complete [ 7779.878831] LustreError: 367506:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 7779.884158] LustreError: 367506:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7783.394111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7783.394703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7783.400502] Lustre: Skipped 1 previous similar message [ 7783.403649] Lustre: Skipped 1 previous similar message [ 7788.514177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7788.517869] Lustre: Skipped 2 previous similar messages [ 7793.633457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7793.636409] Lustre: Skipped 1 previous similar message [ 7795.168123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7795.228673] Lustre: server umount lustre-MDT0000 complete [ 7796.539262] LustreError: 364640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859506 with bad export cookie 589275977384061141 [ 7796.543526] LustreError: 364640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7796.544491] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7807.023363] Lustre: server umount lustre-MDT0001 complete [ 7810.493654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7812.174668] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7813.127281] LustreError: 368647:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7815.174869] Lustre: 368985:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.104@tcp [ 7815.844199] Lustre: server umount MGS complete [ 7819.825393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7819.968147] LustreError: 369710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7819.972312] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7819.973684] LustreError: 369710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7819.975408] Lustre: Skipped 1 previous similar message [ 7819.992774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7821.343187] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7824.470551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7824.563625] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7825.860703] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7827.642304] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7828.801338] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7829.441815] Lustre: Permanently reactivating lustre-OST0000 [ 7829.447549] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7829.450070] Lustre: Skipped 2 previous similar messages [ 7849.952134] LustreError: 371059:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7852.967042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7853.094757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7853.097309] Lustre: Skipped 1 previous similar message [ 7855.046856] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7856.757229] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7869.411324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7869.411983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7869.414198] LustreError: Skipped 1 previous similar message [ 7869.419298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7869.421948] Lustre: Skipped 1 previous similar message [ 7870.929408] Lustre: server umount lustre-OST0000 complete [ 7874.339959] Lustre: server umount lustre-MDT0000 complete [ 7874.530414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.531108] LustreError: 369704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7874.534192] Lustre: Skipped 2 previous similar messages [ 7875.609272] LustreError: 372793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859585 with bad export cookie 589275977384062177 [ 7875.612991] LustreError: 372793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7875.613147] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7875.732701] Lustre: server umount lustre-MDT0001 complete [ 7879.348922] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 7880.302710] Key type lgssc unregistered [ 7880.424272] LNet: 373825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7880.426728] LNetError: 373825:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7880.434289] LNet: Removed LNI 192.168.202.104@tcp [ 7880.723433] Key type .llcrypt unregistered [ 7880.724572] Key type ._llcrypt unregistered [ 7886.598880] Key type ._llcrypt registered [ 7886.599967] Key type .llcrypt registered [ 7886.639779] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 7892.662799] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 7893.058561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7893.114873] alg: No test for adler32 (adler32-zlib) [ 7893.979118] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 7894.066207] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 7895.648204] Key type lgssc registered [ 7896.037472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7907.514231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7907.521863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7908.622837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7908.633934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7908.675497] Lustre: lustre-MDT0000: new disk, initializing [ 7908.700498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7908.706996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7909.985715] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7914.298221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7914.340702] Lustre: 376935:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7914.374282] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7914.376464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7914.379576] Lustre: Skipped 1 previous similar message [ 7914.416681] Lustre: lustre-MDT0001: new disk, initializing [ 7914.449640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7914.463791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7914.468576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7915.739282] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7918.805773] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7920.011076] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7922.312445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7922.416428] Lustre: lustre-OST0000: new disk, initializing [ 7922.417928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7922.420152] Lustre: 378677:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7922.448161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7924.300590] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7924.345852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7924.349151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7924.391532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7927.473211] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7928.862574] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7928.950847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7930.233253] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7930.305667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7934.947719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7934.951033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7934.953215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7934.954884] LustreError: Skipped 1 previous similar message [ 7937.108912] Lustre: server umount lustre-OST0000 complete [ 7940.067043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7940.067502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7940.067986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7940.067991] Lustre: Skipped 1 previous similar message [ 7940.077550] Lustre: Skipped 2 previous similar messages [ 7944.546756] Lustre: server umount lustre-MDT0000 complete [ 7945.813420] LustreError: 377713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859655 with bad export cookie 11347515290041540756 [ 7945.816276] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7945.816558] LustreError: 377713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7945.939221] Lustre: server umount lustre-MDT0001 complete [ 7948.782748] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7949.388574] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:14:20 (1780859660) [ 7951.791810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7951.969382] LustreError: 380987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7951.974986] LustreError: 380987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7951.994317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7953.346940] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7956.797166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7957.001295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7958.628008] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7960.520827] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7961.653075] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7964.354091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7966.286431] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 7968.113362] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7968.778160] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7979.718118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7982.051589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7982.052073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7982.059439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7982.061131] Lustre: Skipped 2 previous similar messages [ 7987.171335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7987.174584] Lustre: Skipped 1 previous similar message [ 7987.343730] Lustre: server umount lustre-OST0000 complete [ 7992.289633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7992.292992] Lustre: Skipped 1 previous similar message [ 7994.712604] Lustre: server umount lustre-MDT0000 complete [ 7995.910196] LustreError: 383350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859705 with bad export cookie 11347515290041541610 [ 7995.913541] LustreError: 383350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7995.913709] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7996.035022] Lustre: server umount lustre-MDT0001 complete [ 7999.628905] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 8000.646734] Key type lgssc unregistered [ 8000.765273] LNet: 384372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8000.767439] LNetError: 384372:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8000.778356] LNet: Removed LNI 192.168.202.104@tcp [ 8001.063968] Key type .llcrypt unregistered [ 8001.066149] Key type ._llcrypt unregistered [ 8008.458911] Key type ._llcrypt registered [ 8008.459964] Key type .llcrypt registered [ 8008.496281] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 8008.596357] Key type .llcrypt unregistered [ 8008.597548] Key type ._llcrypt unregistered [ 8015.294135] Key type ._llcrypt registered [ 8015.295147] Key type .llcrypt registered [ 8015.333436] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 8015.650300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8015.673054] alg: No test for adler32 (adler32-zlib) [ 8016.524293] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 8016.610416] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 8018.192154] Key type lgssc registered [ 8018.549440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8021.094799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8021.100502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8022.258601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8023.505547] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8026.422024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8026.551590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8028.407517] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8030.124330] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8031.714698] LustreError: 386520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8031.719807] LustreError: 386520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8032.425815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8032.567919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8033.576893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8033.869573] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8037.857457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8037.860770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8037.864610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8038.884065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8038.887105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8038.889425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8038.891017] Lustre: Skipped 1 previous similar message [ 8042.765197] Lustre: server umount lustre-OST0000 complete [ 8044.002884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8044.007225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8044.010517] Lustre: Skipped 1 previous similar message [ 8049.121700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8049.123470] Lustre: Skipped 1 previous similar message [ 8050.133990] Lustre: server umount lustre-MDT0000 complete [ 8051.442689] LustreError: 388486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859761 with bad export cookie 4681931774835078101 [ 8051.447354] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8051.448914] LustreError: 388486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8051.577832] Lustre: server umount lustre-MDT0001 complete [ 8055.346417] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 8056.412433] Key type lgssc unregistered [ 8056.560602] LNet: 389511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8056.564621] LNetError: 389511:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8056.572430] LNet: Removed LNI 192.168.202.104@tcp [ 8056.954472] Key type .llcrypt unregistered [ 8056.956355] Key type ._llcrypt unregistered [ 8066.543556] Key type ._llcrypt registered [ 8066.544689] Key type .llcrypt registered [ 8066.579176] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 8067.018392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8067.026464] alg: No test for adler32 (adler32-zlib) [ 8067.932453] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 8068.031689] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 8069.624261] Key type lgssc registered [ 8070.115977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8073.075352] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8073.082601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8088.544407] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8093.664180] LustreError: 390861:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.104@tcp: failed processing log, type 4: rc = -110 [ 8120.288254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8122.306431] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8124.124739] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8126.808733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8127.030867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8127.056139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8128.442745] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8131.326786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8131.515634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8132.860604] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8134.780793] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8136.001826] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8139.749720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8139.751666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8139.753620] Lustre: Skipped 1 previous similar message [ 8139.756828] Lustre: Skipped 1 previous similar message [ 8144.533596] Lustre: server umount lustre-OST0000 complete [ 8146.913198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8146.915652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8146.920072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8149.985627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8149.986383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8151.900666] Lustre: server umount lustre-MDT0000 complete [ 8153.460470] LustreError: 393049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859863 with bad export cookie 3514679923830673173 [ 8153.464533] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8153.464843] LustreError: 393049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8153.590588] Lustre: server umount lustre-MDT0001 complete [ 8157.715175] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 8158.913980] Key type lgssc unregistered [ 8159.045529] LNet: 394072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8159.049250] LNetError: 394072:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8159.058406] LNet: Removed LNI 192.168.202.104@tcp [ 8159.388282] Key type .llcrypt unregistered [ 8159.389363] Key type ._llcrypt unregistered [ 8168.788887] Key type ._llcrypt registered [ 8168.789877] Key type .llcrypt registered [ 8168.829937] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 8169.277846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8169.346597] alg: No test for adler32 (adler32-zlib) [ 8170.250020] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 8170.345044] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 8171.928299] Key type lgssc registered [ 8172.380315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8175.266256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8175.276240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8176.474523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8177.897562] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8180.757471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.895201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8182.220400] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8184.018689] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8185.326806] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8187.591171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8187.741665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.673194] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8191.421420] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8202.090261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8211.937089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8211.940103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8211.944565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8212.449662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8212.450876] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8212.451536] Lustre: Skipped 1 previous similar message [ 8212.454503] Lustre: Skipped 1 previous similar message [ 8213.987274] Lustre: server umount lustre-MDT0000 complete [ 8214.702532] LustreError: 395429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.4@tcp arrived at 1780859924 with bad export cookie 9360496887432663653 [ 8214.706376] LustreError: 395429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8215.334243] LustreError: 395430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859925 with bad export cookie 9360496887432663030 [ 8215.337979] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8215.340553] LustreError: 395430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8216.545120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8216.548350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8216.553892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8216.556365] Lustre: Skipped 1 previous similar message [ 8217.568965] LustreError: 395449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8217.573335] LustreError: 395449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8220.576845] LustreError: 395450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8220.577247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8220.582546] LustreError: 395450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8220.585940] Lustre: Skipped 1 previous similar message [ 8221.454664] Lustre: server umount lustre-OST0000 complete [ 8223.709927] Lustre: server umount lustre-MDT0001 complete [ 8227.398359] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 8228.377914] Key type lgssc unregistered [ 8228.509348] LNet: 398797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8228.511624] LNetError: 398797:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8228.518277] LNet: Removed LNI 192.168.202.104@tcp [ 8228.813731] Key type .llcrypt unregistered [ 8228.814780] Key type ._llcrypt unregistered [ 8238.123259] Key type ._llcrypt registered [ 8238.124479] Key type .llcrypt registered [ 8238.164262] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 8238.525552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8238.540314] alg: No test for adler32 (adler32-zlib) [ 8239.414288] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 8239.508509] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 8241.096216] Key type lgssc registered [ 8241.489044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8244.070849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8244.077954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8245.263774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8246.862135] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8249.774151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8249.904455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8251.192951] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8253.065921] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8254.271603] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8256.520903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8256.641432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.534474] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8260.309033] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8271.018772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8280.544876] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8280.547376] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8280.551540] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8281.217158] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.4@tcp (stopping) [ 8281.571174] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8283.285218] Lustre: server umount lustre-MDT0001 complete [ 8286.689677] LustreError: 400173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8286.690228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8286.690651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8286.691333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8286.691338] Lustre: Skipped 2 previous similar messages [ 8286.694908] LustreError: 400173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8291.025260] Lustre: server umount lustre-OST0000 complete [ 8291.809451] LustreError: 400897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8292.401772] Lustre: server umount lustre-MDT0000 complete [ 8296.812246] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 8297.924342] Key type lgssc unregistered [ 8298.065517] LNet: 403522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8298.068321] LNetError: 403522:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8298.075460] LNet: Removed LNI 192.168.202.104@tcp [ 8298.396796] Key type .llcrypt unregistered [ 8298.397894] Key type ._llcrypt unregistered [ 8307.802416] Key type ._llcrypt registered [ 8307.803664] Key type .llcrypt registered [ 8307.847068] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 8314.318334] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 8314.692285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8314.742381] alg: No test for adler32 (adler32-zlib) [ 8315.605833] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 8315.701392] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 8317.288162] Key type lgssc registered [ 8317.712968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8332.485238] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 8338.068305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8338.074960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8339.170891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8339.182087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8339.219310] Lustre: lustre-MDT0000: new disk, initializing [ 8339.245563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8339.252235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8340.599598] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8346.089203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8346.189566] Lustre: lustre-OST0000: new disk, initializing [ 8346.191071] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8346.192480] Lustre: Skipped 1 previous similar message [ 8346.194191] Lustre: 408459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8346.217240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8348.051362] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8351.076768] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8353.789500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8353.792446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8353.799647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8361.781398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8361.828115] Lustre: 407458:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8361.865753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8361.906241] Lustre: lustre-MDT0001: new disk, initializing [ 8361.956725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8361.972122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8361.976573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8363.313967] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8366.859376] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8366.946254] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8368.270554] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8370.377463] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 8372.352827] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.4@tcp (stopping) [ 8374.754207] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.754279] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8374.758318] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8374.763364] Lustre: Skipped 1 previous similar message [ 8377.467342] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.4@tcp (stopping) [ 8378.007539] Lustre: server umount lustre-MDT0001 complete [ 8384.995277] LustreError: 407465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8385.001853] LustreError: 407465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8390.116068] LustreError: 407463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8390.122995] LustreError: 407463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8394.721243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8394.723382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8394.726610] Lustre: Skipped 1 previous similar message [ 8394.728303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8395.234171] LustreError: 407469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8396.499728] Lustre: server umount lustre-OST0000 complete [ 8412.128101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8412.218731] Lustre: server umount lustre-MDT0000 complete [ 8417.184556] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 8418.288935] Key type lgssc unregistered [ 8418.416390] LNet: 412040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8418.419610] LNetError: 412040:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8418.429465] LNet: Removed LNI 192.168.202.104@tcp [ 8418.744220] Key type .llcrypt unregistered [ 8418.745778] Key type ._llcrypt unregistered [ 8431.566277] Key type ._llcrypt registered [ 8431.567329] Key type .llcrypt registered [ 8431.610261] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 8431.720213] Key type .llcrypt unregistered [ 8431.721201] Key type ._llcrypt unregistered [ 8438.341897] Key type ._llcrypt registered [ 8438.343014] Key type .llcrypt registered [ 8438.382741] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 8438.931713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8438.947304] alg: No test for adler32 (adler32-zlib) [ 8439.815904] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 8439.920758] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 8441.512130] Key type lgssc registered [ 8441.875865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8450.540645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8450.545474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8451.646414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8451.656440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8451.696244] Lustre: lustre-MDT0000: new disk, initializing [ 8451.720950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8451.727535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8452.925674] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8457.219823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8457.267429] Lustre: 415352:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8457.299961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8457.301984] Lustre: Skipped 1 previous similar message [ 8457.342523] Lustre: lustre-MDT0001: new disk, initializing [ 8457.383129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8457.397788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8457.401221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8458.740484] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8461.761686] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8462.928993] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8465.102090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8465.214134] Lustre: lustre-OST0000: new disk, initializing [ 8465.215945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8465.218553] Lustre: 417093:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8465.250275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8467.131231] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8470.146678] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8470.542553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8470.549408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8470.599479] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8478.177921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8478.181309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8478.187131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8480.740505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8480.741127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8480.745569] Lustre: Skipped 1 previous similar message [ 8480.756322] Lustre: Skipped 1 previous similar message [ 8484.327795] Lustre: server umount lustre-MDT0000 complete [ 8485.916904] LustreError: 415347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780860195 with bad export cookie 15225365982182222972 [ 8485.923086] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8485.924535] LustreError: 415347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8486.081911] Lustre: server umount lustre-MDT0001 complete [ 8497.942729] Lustre: server umount lustre-OST0000 complete [ 8502.126586] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:23:32 (1780860212) [ 8505.921360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8506.076102] LustreError: 419007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8506.081075] LustreError: 419007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8506.099740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8507.857612] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8511.215263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8511.349479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8512.692717] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8515.369956] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8516.719564] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8532.194919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8532.280112] Lustre: Found index 0 for lustre-OST0000, updating log [ 8532.354783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8534.252491] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 8536.206937] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8552.877978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8562.145219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8562.149255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8562.156331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8564.716445] Lustre: server umount lustre-OST0000 complete [ 8577.304783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8577.305447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8577.308210] Lustre: Skipped 2 previous similar messages [ 8577.309812] Lustre: Skipped 3 previous similar messages [ 8582.624980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8582.627068] Lustre: Skipped 1 previous similar message [ 8583.450581] Lustre: server umount lustre-MDT0000 complete [ 8584.655381] LustreError: 418988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780860294 with bad export cookie 15225365982182224351 [ 8584.659817] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8584.770599] Lustre: server umount lustre-MDT0001 complete [ 8587.566930] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:24:58 (1780860298) [ 8600.672334] LNet: 1 local NIs in recovery (showing 1): 192.168.202.104@tcp [ 9170.002397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9170.204532] LustreError: 425092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9170.240596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9171.829752] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9175.133352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9175.293746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9176.683227] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9178.639404] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9180.052994] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9195.556885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9195.697966] Lustre: Found index 0 for lustre-OST0000, updating log [ 9195.829745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9197.868917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 9198.201922] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9200.730360] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9202.594682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9202.600613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9207.705175] Lustre: server umount lustre-OST0000 complete [ 9210.082553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9210.083895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9210.087972] Lustre: Skipped 1 previous similar message [ 9212.130701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9212.132543] Lustre: Skipped 3 previous similar messages [ 9215.198469] Lustre: server umount lustre-MDT0000 complete [ 9215.203580] LustreError: 425093:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9215.212564] LustreError: 425093:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9216.225619] LustreError: 425804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9216.232163] LustreError: 425804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9216.638456] LustreError: 426778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780860926 with bad export cookie 15225365982182225422 [ 9216.643372] LustreError: 426778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9216.644052] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9216.789270] Lustre: server umount lustre-MDT0001 complete [ 9228.384405] LNet: 1 local NIs in recovery (showing 1): 192.168.202.104@tcp [ 9235.986717] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 9239.737834] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9252.437378] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:36:03 (1780860963) [ 9258.004650] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 9259.145196] Key type lgssc unregistered [ 9259.285976] LNet: 431355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9259.290899] LNetError: 431355:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9327.910644] LNet: Removed LNI 192.168.202.104@tcp [ 9328.440720] Key type .llcrypt unregistered [ 9328.443054] Key type ._llcrypt unregistered [ 9336.720342] Key type ._llcrypt registered [ 9336.722339] Key type .llcrypt registered [ 9336.793962] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9337.379088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9337.387619] alg: No test for adler32 (adler32-zlib) [ 9338.295663] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 9338.401209] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 9340.000263] Key type lgssc registered [ 9340.675195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9342.651627] LNet: Added LNI 192.168.202.104@tcp100 [8/256/0/180] [ 9343.031114] LNet: Added LNI 192.168.202.104@tcp101 [8/256/0/180] [ 9343.431816] LNet: Added LNI 192.168.202.104@tcp102 [8/256/0/180] [ 9343.831021] LNet: Added LNI 192.168.202.104@tcp103 [8/256/0/180] [ 9344.211923] LNet: Added LNI 192.168.202.104@tcp104 [8/256/0/180] [ 9344.672749] LNet: Added LNI 192.168.202.104@tcp105 [8/256/0/180] [ 9345.115441] LNet: Added LNI 192.168.202.104@tcp106 [8/256/0/180] [ 9345.577676] LNet: Added LNI 192.168.202.104@tcp107 [8/256/0/180] [ 9345.984807] LNet: Added LNI 192.168.202.104@tcp108 [8/256/0/180] [ 9346.391430] LNet: Added LNI 192.168.202.104@tcp109 [8/256/0/180] [ 9346.728809] LNet: Added LNI 192.168.202.104@tcp110 [8/256/0/180] [ 9347.113703] LNet: Added LNI 192.168.202.104@tcp111 [8/256/0/180] [ 9347.455391] LNet: Added LNI 192.168.202.104@tcp112 [8/256/0/180] [ 9347.810076] LNet: Added LNI 192.168.202.104@tcp113 [8/256/0/180] [ 9348.256451] LNet: Added LNI 192.168.202.104@tcp114 [8/256/0/180] [ 9348.663753] LNet: Added LNI 192.168.202.104@tcp115 [8/256/0/180] [ 9349.045284] LNet: Added LNI 192.168.202.104@tcp116 [8/256/0/180] [ 9349.412290] LNet: Added LNI 192.168.202.104@tcp117 [8/256/0/180] [ 9349.726657] LNet: Added LNI 192.168.202.104@tcp118 [8/256/0/180] [ 9350.000809] LNet: Added LNI 192.168.202.104@tcp119 [8/256/0/180] [ 9350.333921] LNet: Added LNI 192.168.202.104@tcp120 [8/256/0/180] [ 9350.721381] LNet: Added LNI 192.168.202.104@tcp121 [8/256/0/180] [ 9351.123224] LNet: Added LNI 192.168.202.104@tcp122 [8/256/0/180] [ 9351.560588] LNet: Added LNI 192.168.202.104@tcp123 [8/256/0/180] [ 9351.992810] LNet: Added LNI 192.168.202.104@tcp124 [8/256/0/180] [ 9352.420245] LNet: Added LNI 192.168.202.104@tcp125 [8/256/0/180] [ 9352.863431] LNet: Added LNI 192.168.202.104@tcp126 [8/256/0/180] [ 9353.312972] LNet: Added LNI 192.168.202.104@tcp127 [8/256/0/180] [ 9353.734966] LNet: Added LNI 192.168.202.104@tcp128 [8/256/0/180] [ 9354.137621] LNet: Added LNI 192.168.202.104@tcp129 [8/256/0/180] [ 9354.545288] LNet: Added LNI 192.168.202.104@tcp130 [8/256/0/180] [ 9354.947721] LNet: Added LNI 192.168.202.104@tcp131 [8/256/0/180] [ 9355.378509] LNet: Added LNI 192.168.202.104@tcp132 [8/256/0/180] [ 9355.824645] LNet: Added LNI 192.168.202.104@tcp133 [8/256/0/180] [ 9356.282220] LNet: Added LNI 192.168.202.104@tcp134 [8/256/0/180] [ 9356.736798] LNet: Added LNI 192.168.202.104@tcp135 [8/256/0/180] [ 9359.819581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9359.827369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9360.896313] Lustre: 434335:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.104@tcp: can't add NID 192.168.202.104@tcp130: rc = -75 [ 9360.995315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9361.006111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9361.051465] Lustre: lustre-MDT0000: new disk, initializing [ 9361.086665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9361.093582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9361.954156] LNetError: 434356:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.104 [ 9361.960635] LNetError: Protocol error connecting to 192.168.202.104@tcp115 at host 192.168.202.104:988: is it running a compatible version of Lustre? [ 9362.819670] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9368.233052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9368.329065] Lustre: 434353:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9368.386490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9368.390425] Lustre: Skipped 1 previous similar message [ 9368.438515] Lustre: lustre-MDT0001: new disk, initializing [ 9368.454293] Lustre: 435136:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.104@tcp: can't add NID 192.168.202.104@tcp130: rc = -75 [ 9368.457763] Lustre: 435136:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9368.512092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9368.536044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9368.540683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9370.136076] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9373.171673] Lustre: 435594:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.104@tcp: can't add NID 192.168.202.104@tcp130: rc = -75 [ 9373.174480] Lustre: 435594:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9373.475452] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9375.051277] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9377.713024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9377.851969] Lustre: lustre-OST0000: new disk, initializing [ 9377.854429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9377.857903] Lustre: 436100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9377.898465] Lustre: 436105:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.104@tcp: can't add NID 192.168.202.104@tcp130: rc = -75 [ 9377.902980] Lustre: 436105:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 9377.921995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9380.215910] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9381.345195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9381.352293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9381.443565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9383.747845] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9391.591601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9391.592951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9391.596208] LustreError: Skipped 1 previous similar message [ 9391.597188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9391.603970] Lustre: Skipped 1 previous similar message [ 9395.745895] LNetError: 434355:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.104 [ 9395.752614] LNetError: Protocol error connecting to 192.168.202.104@tcp115 at host 192.168.202.104:988: is it running a compatible version of Lustre? [ 9396.707894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9396.711340] Lustre: Skipped 2 previous similar messages [ 9401.312197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9401.379557] Lustre: server umount lustre-OST0000 complete [ 9404.385521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9404.390468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9404.399198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9406.946729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9406.947689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9408.033516] LNetError: 431798:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.104 [ 9408.037649] LNetError: Protocol error connecting to 192.168.202.104@tcp115 at host 192.168.202.104:988: is it running a compatible version of Lustre? [ 9409.131823] Lustre: server umount lustre-MDT0000 complete [ 9410.433595] LustreError: 434344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780861120 with bad export cookie 4886972051463785705 [ 9410.436407] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9410.440384] LustreError: 434344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9410.585817] Lustre: server umount lustre-MDT0001 complete [ 9414.323896] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 9415.976933] Key type lgssc unregistered [ 9416.229291] LNet: 437885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9416.238459] LNetError: 437885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9416.244089] LNet: Removed LNI 192.168.202.104@tcp [ 9417.248170] LNet: Removed LNI 192.168.202.104@tcp100 [ 9417.249695] LNetError: 437885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9417.251858] LNetError: 437885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9417.254232] LNet: Removed LNI 192.168.202.104@tcp101 [ 9418.272221] LNet: Removed LNI 192.168.202.104@tcp102 [ 9418.275107] LNetError: 437885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9418.279925] LNetError: 437885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9418.285045] LNet: Removed LNI 192.168.202.104@tcp103 [ 9419.296239] LNet: Removed LNI 192.168.202.104@tcp104 [ 9420.320230] LNet: Removed LNI 192.168.202.104@tcp105 [ 9420.323148] LNetError: 437885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9420.327798] LNetError: 437885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9420.332101] LNet: Removed LNI 192.168.202.104@tcp106 [ 9421.344160] LNet: Removed LNI 192.168.202.104@tcp107 [ 9422.368226] LNet: Removed LNI 192.168.202.104@tcp108 [ 9423.392156] LNet: Removed LNI 192.168.202.104@tcp109 [ 9424.416126] LNet: Removed LNI 192.168.202.104@tcp110 [ 9424.417350] LNetError: 437885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9424.419410] LNetError: 437885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9424.421542] LNet: Removed LNI 192.168.202.104@tcp111 [ 9425.440112] LNet: Removed LNI 192.168.202.104@tcp112 [ 9426.464116] LNet: Removed LNI 192.168.202.104@tcp113 [ 9427.488231] LNet: Removed LNI 192.168.202.104@tcp114 [ 9428.512128] LNet: Removed LNI 192.168.202.104@tcp115 [ 9429.536131] LNet: Removed LNI 192.168.202.104@tcp116 [ 9429.538866] LNet: Removed LNI 192.168.202.104@tcp117 [ 9429.540272] LNet: Removed LNI 192.168.202.104@tcp118 [ 9429.541585] LNet: Removed LNI 192.168.202.104@tcp119 [ 9429.542953] LNet: Removed LNI 192.168.202.104@tcp120 [ 9429.544067] LNet: Removed LNI 192.168.202.104@tcp121 [ 9429.545273] LNet: Removed LNI 192.168.202.104@tcp122 [ 9429.546466] LNet: Removed LNI 192.168.202.104@tcp123 [ 9429.547683] LNet: Removed LNI 192.168.202.104@tcp124 [ 9429.548899] LNet: Removed LNI 192.168.202.104@tcp125 [ 9429.550098] LNet: Removed LNI 192.168.202.104@tcp126 [ 9429.551280] LNet: Removed LNI 192.168.202.104@tcp127 [ 9429.552565] LNet: Removed LNI 192.168.202.104@tcp128 [ 9429.553678] LNet: Removed LNI 192.168.202.104@tcp129 [ 9429.554849] LNet: Removed LNI 192.168.202.104@tcp130 [ 9429.556031] LNet: Removed LNI 192.168.202.104@tcp131 [ 9429.557267] LNet: Removed LNI 192.168.202.104@tcp132 [ 9429.558418] LNet: Removed LNI 192.168.202.104@tcp133 [ 9429.559718] LNet: Removed LNI 192.168.202.104@tcp134 [ 9429.568820] LNet: Removed LNI 192.168.202.104@tcp135 [ 9429.872504] Key type .llcrypt unregistered [ 9429.873726] Key type ._llcrypt unregistered [ 9439.935295] Key type ._llcrypt registered [ 9439.936538] Key type .llcrypt registered [ 9439.984029] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 9446.962798] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9447.445270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9447.454053] alg: No test for adler32 (adler32-zlib) [ 9448.341603] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 9448.450226] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 9450.032159] Key type lgssc registered [ 9450.486783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9463.707795] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:39:34 (1780861174) [ 9471.591762] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 9473.120507] Key type lgssc unregistered [ 9473.281689] LNet: 441892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9473.287563] LNetError: 441892:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9473.302566] LNet: Removed LNI 192.168.202.104@tcp [ 9473.749686] Key type .llcrypt unregistered [ 9473.752176] Key type ._llcrypt unregistered [ 9479.933115] Key type ._llcrypt registered [ 9479.934317] Key type .llcrypt registered [ 9479.974638] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9480.377997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9480.385325] alg: No test for adler32 (adler32-zlib) [ 9481.238329] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 9481.326200] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 9482.904126] Key type lgssc registered [ 9483.227786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9486.656879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9486.667197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9487.777883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9487.788977] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9487.831915] Lustre: lustre-MDT0000: new disk, initializing [ 9487.860375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9487.871725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9489.315242] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9494.132641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9494.200916] Lustre: 443066:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9494.233921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9494.237268] Lustre: Skipped 1 previous similar message [ 9494.285967] Lustre: lustre-MDT0001: new disk, initializing [ 9494.328808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9494.347797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9494.353798] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9495.942541] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9499.592713] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9501.073522] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9504.270595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9504.431542] Lustre: lustre-OST0000: new disk, initializing [ 9504.433866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9504.437230] Lustre: 444808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9504.472862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9507.004422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9507.004797] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9507.008134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9507.042771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9510.460580] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9511.792369] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing lnet_if_list [ 9513.624278] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9513.635995] LNet: Added LNI 192.168.202.104@tcp10 [8/256/0/180] [ 9515.616469] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9515.628453] LNet: Added LNI 192.168.202.104@tcp11 [8/256/0/180] [ 9517.124162] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9517.139516] LNet: Added LNI 192.168.202.104@tcp12 [8/256/0/180] [ 9518.795332] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9518.806000] LNet: Added LNI 192.168.202.104@tcp13 [8/256/0/180] [ 9520.344236] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9520.361336] LNet: Added LNI 192.168.202.104@tcp14 [8/256/0/180] [ 9522.323810] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9522.343464] LNet: Added LNI 192.168.202.104@tcp15 [8/256/0/180] [ 9524.150680] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9524.162435] LNet: Added LNI 192.168.202.104@tcp16 [8/256/0/180] [ 9525.925271] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9525.939073] LNet: Added LNI 192.168.202.104@tcp17 [8/256/0/180] [ 9527.926522] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9527.949607] LNet: Added LNI 192.168.202.104@tcp18 [8/256/0/180] [ 9530.177941] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9530.197247] LNet: Added LNI 192.168.202.104@tcp19 [8/256/0/180] [ 9532.083196] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9532.097294] LNet: Added LNI 192.168.202.104@tcp20 [8/256/0/180] [ 9534.307133] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9534.328155] LNet: Added LNI 192.168.202.104@tcp21 [8/256/0/180] [ 9536.500754] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9536.516226] LNet: Added LNI 192.168.202.104@tcp22 [8/256/0/180] [ 9538.804220] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9538.822957] LNet: Added LNI 192.168.202.104@tcp23 [8/256/0/180] [ 9541.146598] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9541.171167] LNet: Added LNI 192.168.202.104@tcp24 [8/256/0/180] [ 9543.710245] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9543.730719] LNet: Added LNI 192.168.202.104@tcp25 [8/256/0/180] [ 9546.003777] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9546.027727] LNet: Added LNI 192.168.202.104@tcp26 [8/256/0/180] [ 9548.797189] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9548.821587] LNet: Added LNI 192.168.202.104@tcp27 [8/256/0/180] [ 9551.028906] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9551.040427] LNet: Added LNI 192.168.202.104@tcp28 [8/256/0/180] [ 9553.715905] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9553.737989] LNet: Added LNI 192.168.202.104@tcp29 [8/256/0/180] [ 9556.665356] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9556.688820] LNet: Added LNI 192.168.202.104@tcp30 [8/256/0/180] [ 9559.592104] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9559.627561] LNet: Added LNI 192.168.202.104@tcp31 [8/256/0/180] [ 9562.529354] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9562.541265] LNet: Added LNI 192.168.202.104@tcp32 [8/256/0/180] [ 9565.241100] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9565.263481] LNet: Added LNI 192.168.202.104@tcp33 [8/256/0/180] [ 9568.346960] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 9568.364796] LNet: Added LNI 192.168.202.104@tcp34 [8/256/0/180] [ 9571.598217] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 9571.611078] LNet: Added LNI 192.168.202.104@tcp35 [8/256/0/180] [ 9574.584514] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 9574.602993] LNet: Added LNI 192.168.202.104@tcp36 [8/256/0/180] [ 9577.319310] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 9577.331157] LNet: Added LNI 192.168.202.104@tcp37 [8/256/0/180] [ 9579.754515] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 9579.767475] LNet: Added LNI 192.168.202.104@tcp38 [8/256/0/180] [ 9582.679973] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 9582.695942] LNet: Added LNI 192.168.202.104@tcp39 [8/256/0/180] [ 9585.397129] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 9585.412268] LNet: Added LNI 192.168.202.104@tcp40 [8/256/0/180] [ 9588.663202] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 9588.675336] LNet: Added LNI 192.168.202.104@tcp41 [8/256/0/180] [ 9592.152885] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 9592.172474] LNet: Added LNI 192.168.202.104@tcp42 [8/256/0/180] [ 9595.752610] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9595.781394] LNet: Added LNI 192.168.202.104@tcp43 [8/256/0/180] [ 9599.372820] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9599.388912] LNet: Added LNI 192.168.202.104@tcp44 [8/256/0/180] [ 9602.921777] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9602.942002] LNet: Added LNI 192.168.202.104@tcp45 [8/256/0/180] [ 9606.493374] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9606.512768] LNet: Added LNI 192.168.202.104@tcp46 [8/256/0/180] [ 9609.866412] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9609.876143] LNet: Added LNI 192.168.202.104@tcp47 [8/256/0/180] [ 9612.228032] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9612.247805] LNet: Added LNI 192.168.202.104@tcp48 [8/256/0/180] [ 9615.691133] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9615.702370] LNet: Added LNI 192.168.202.104@tcp49 [8/256/0/180] [ 9619.539532] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9619.552729] LNet: Added LNI 192.168.202.104@tcp50 [8/256/0/180] [ 9630.179907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9630.185201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9630.188347] Lustre: Skipped 2 previous similar messages [ 9630.193507] Lustre: Skipped 1 previous similar message [ 9634.825877] Lustre: server umount lustre-MDT0000 complete [ 9636.657491] LustreError: 443059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780861346 with bad export cookie 9776398094597624772 [ 9636.663483] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9636.665507] LustreError: 443059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9636.826919] Lustre: server umount lustre-MDT0001 complete [ 9644.483502] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9648.443664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9656.800303] Lustre: 442341:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780861350/real 1780861350] req@00000000568ac3ef x1867368304308096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780861366 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9656.820217] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9657.824143] Lustre: 442338:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780861351/real 1780861351] req@0000000098f9994b x1867368304308224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780861367 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9661.920206] Lustre: 442339:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780861355/real 1780861355] req@000000003c8d853a x1867368304308352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780861371 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9661.984510] LustreError: 442337:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000aa29616 x1867368304309248/t0(0) o250->MGC192.168.202.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9662.136851] LustreError: 444800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9662.143718] LustreError: 444800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9662.157779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9663.980760] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9665.187532] LustreError: 444801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9666.212034] LustreError: 444800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9666.221798] LustreError: 444800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9667.232290] Lustre: 442341:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780861361/real 1780861361] req@000000008f2a90e4 x1867368304308736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780861377 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9667.240932] Lustre: 442341:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9667.859251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9668.017461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9669.781398] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9671.527364] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9672.097112] Lustre: 442339:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780861366/real 1780861366] req@00000000070e6825 x1867368304308864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780861382 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9673.144068] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9673.194301] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9682.913285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9682.915905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9682.920796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9682.922853] Lustre: Skipped 1 previous similar message [ 9683.430284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9688.490718] Lustre: server umount lustre-OST0000 complete [ 9693.667199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9693.667384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9693.669161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9693.669170] Lustre: Skipped 2 previous similar messages [ 9693.676042] Lustre: Skipped 2 previous similar messages [ 9696.502421] Lustre: server umount lustre-MDT0000 complete [ 9698.366382] LustreError: 443058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780861408 with bad export cookie 9776398094597626893 [ 9698.376053] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9698.525436] Lustre: server umount lustre-MDT0001 complete [ 9703.160784] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 9704.777906] Key type lgssc unregistered [ 9704.994719] LNet: 470684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9705.001438] LNetError: 470684:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9705.006280] LNet: Removed LNI 192.168.202.104@tcp [ 9705.008648] LNet: Removed LNI 192.168.202.104@tcp10 [ 9705.010907] LNet: Removed LNI 192.168.202.104@tcp11 [ 9705.013245] LNet: Removed LNI 192.168.202.104@tcp12 [ 9705.015516] LNet: Removed LNI 192.168.202.104@tcp13 [ 9705.017731] LNet: Removed LNI 192.168.202.104@tcp14 [ 9705.019994] LNet: Removed LNI 192.168.202.104@tcp15 [ 9705.022272] LNet: Removed LNI 192.168.202.104@tcp16 [ 9705.024450] LNet: Removed LNI 192.168.202.104@tcp17 [ 9705.026566] LNet: Removed LNI 192.168.202.104@tcp18 [ 9705.028661] LNet: Removed LNI 192.168.202.104@tcp19 [ 9705.030749] LNet: Removed LNI 192.168.202.104@tcp20 [ 9705.033651] LNet: Removed LNI 192.168.202.104@tcp21 [ 9705.035672] LNet: Removed LNI 192.168.202.104@tcp22 [ 9705.037779] LNet: Removed LNI 192.168.202.104@tcp23 [ 9705.039686] LNet: Removed LNI 192.168.202.104@tcp24 [ 9705.041583] LNet: Removed LNI 192.168.202.104@tcp25 [ 9705.043411] LNet: Removed LNI 192.168.202.104@tcp26 [ 9705.045360] LNet: Removed LNI 192.168.202.104@tcp27 [ 9705.047271] LNet: Removed LNI 192.168.202.104@tcp28 [ 9705.049221] LNet: Removed LNI 192.168.202.104@tcp29 [ 9705.051077] LNet: Removed LNI 192.168.202.104@tcp30 [ 9705.052823] LNet: Removed LNI 192.168.202.104@tcp31 [ 9705.054641] LNet: Removed LNI 192.168.202.104@tcp32 [ 9705.056361] LNet: Removed LNI 192.168.202.104@tcp33 [ 9705.058046] LNet: Removed LNI 192.168.202.104@tcp34 [ 9705.059615] LNet: Removed LNI 192.168.202.104@tcp35 [ 9705.061335] LNet: Removed LNI 192.168.202.104@tcp36 [ 9705.063125] LNet: Removed LNI 192.168.202.104@tcp37 [ 9705.064795] LNet: Removed LNI 192.168.202.104@tcp38 [ 9705.066480] LNet: Removed LNI 192.168.202.104@tcp39 [ 9705.068088] LNet: Removed LNI 192.168.202.104@tcp40 [ 9705.069583] LNet: Removed LNI 192.168.202.104@tcp41 [ 9705.071912] LNet: Removed LNI 192.168.202.104@tcp42 [ 9705.074231] LNet: Removed LNI 192.168.202.104@tcp43 [ 9705.075857] LNet: Removed LNI 192.168.202.104@tcp44 [ 9705.077303] LNet: Removed LNI 192.168.202.104@tcp45 [ 9705.078648] LNet: Removed LNI 192.168.202.104@tcp46 [ 9705.080131] LNet: Removed LNI 192.168.202.104@tcp47 [ 9705.081496] LNet: Removed LNI 192.168.202.104@tcp48 [ 9705.082906] LNet: Removed LNI 192.168.202.104@tcp49 [ 9705.089791] LNet: Removed LNI 192.168.202.104@tcp50 [ 9705.477155] Key type .llcrypt unregistered [ 9705.479456] Key type ._llcrypt unregistered [ 9714.897912] Key type ._llcrypt registered [ 9714.900280] Key type .llcrypt registered [ 9714.970717] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 9722.669857] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9723.061338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9723.109988] alg: No test for adler32 (adler32-zlib) [ 9723.987142] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 9724.089275] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 9725.696121] Key type lgssc registered [ 9726.090136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9742.033799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9742.040216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9743.128362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9743.139893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9743.179607] Lustre: lustre-MDT0000: new disk, initializing [ 9743.204211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9743.211437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9744.522265] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9749.993123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9750.061517] Lustre: 473820:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9750.096715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9750.100845] Lustre: Skipped 1 previous similar message [ 9750.146993] Lustre: lustre-MDT0001: new disk, initializing [ 9750.188232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9750.206428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9750.211213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9751.861791] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9755.755852] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9756.979875] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9760.401195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9760.513744] Lustre: *** cfs_fail_loc=90f, val=0*** [ 9760.563092] Lustre: lustre-OST0000: new disk, initializing [ 9760.566560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9760.571675] Lustre: 475611:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9760.624987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9763.039353] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9766.699590] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9769.471549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9769.475691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9769.530131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9774.566381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9774.567801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9774.574268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9774.576663] LustreError: Skipped 1 previous similar message [ 9774.583373] Lustre: Skipped 1 previous similar message [ 9776.169293] Lustre: server umount lustre-OST0000 complete [ 9779.683644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9779.684990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9779.689164] Lustre: Skipped 2 previous similar messages [ 9779.693622] Lustre: Skipped 1 previous similar message [ 9784.332583] Lustre: server umount lustre-MDT0000 complete [ 9786.049088] LustreError: 473812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780861495 with bad export cookie 8542909877193409468 [ 9786.052027] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9786.052153] LustreError: 473812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9786.176528] Lustre: server umount lustre-MDT0001 complete [ 9790.274226] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 9791.155771] Key type lgssc unregistered [ 9791.268312] LNet: 477385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9791.271099] LNetError: 477385:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9791.277265] LNet: Removed LNI 192.168.202.104@tcp [ 9791.533992] Key type .llcrypt unregistered [ 9791.535249] Key type ._llcrypt unregistered [ 9800.840221] Key type ._llcrypt registered [ 9800.841536] Key type .llcrypt registered [ 9800.896622] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9801.348678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9801.363707] alg: No test for adler32 (adler32-zlib) [ 9802.204996] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 9802.279948] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 9803.856277] Key type lgssc registered [ 9804.459063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9807.724514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9807.732744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9808.954495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9810.784857] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9815.141338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9815.345751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9817.076618] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9819.664203] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9821.368766] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9824.786328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9825.001255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9827.640849] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9830.152169] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9839.953470] loop0: detected capacity change from 0 to 204800000 [ 9840.024107] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9840.028407] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9840.033852] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9842.830661] loop0: detected capacity change from 0 to 204800000 [ 9842.943714] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9842.952320] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9842.959441] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9845.330340] loop0: detected capacity change from 0 to 204800000 [ 9847.251991] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9847.311241] Lustre: fs15246-OST0000: new disk, initializing [ 9847.313849] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 9847.317615] Lustre: 481869:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9847.340084] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9849.702895] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9854.458733] loop1: detected capacity change from 0 to 204800000 [ 9856.999689] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9857.059071] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 9857.098859] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 9857.154566] Lustre: fs15246-MDT0000: new disk, initializing [ 9857.239359] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9857.255530] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9857.318193] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9858.336721] Lustre: 482973:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000827d01dc x1867368640840960/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780861584 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 9858.895645] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9863.395225] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9863.402166] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9863.412313] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9872.877580] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9872.887103] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9875.707165] Lustre: server umount fs15246-MDT0000 complete [ 9877.991592] LustreError: 483481:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9878.001215] LustreError: 483481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9879.772856] Lustre: server umount fs15246-OST0000 complete [ 9887.713743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9887.719923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9887.729558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9890.862452] Lustre: server umount lustre-OST0000 complete [ 9893.348762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9893.350475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9893.354932] Lustre: Skipped 2 previous similar messages [ 9893.363481] Lustre: Skipped 3 previous similar messages [ 9898.466669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9898.470683] Lustre: Skipped 1 previous similar message [ 9903.585213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9903.587728] Lustre: Skipped 1 previous similar message [ 9907.168257] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9907.258774] Lustre: server umount lustre-MDT0000 complete [ 9908.705825] LustreError: 478615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9908.711179] LustreError: 478615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9908.898786] LustreError: 478596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780861618 with bad export cookie 8105688955903010291 [ 9908.903216] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9908.908385] LustreError: 478596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9909.058331] Lustre: server umount lustre-MDT0001 complete [ 9912.414542] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 9913.826604] Key type lgssc unregistered [ 9913.986535] LNet: 486032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9913.991480] LNetError: 486032:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9914.004601] LNet: Removed LNI 192.168.202.104@tcp [ 9914.474143] Key type .llcrypt unregistered [ 9914.476619] Key type ._llcrypt unregistered [ 9922.650417] Key type ._llcrypt registered [ 9922.651915] Key type .llcrypt registered [ 9922.695324] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9923.007742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9923.049298] alg: No test for adler32 (adler32-zlib) [ 9923.906403] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 9924.003341] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 9925.608196] Key type lgssc registered [ 9926.128878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9944.643869] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 9948.786544] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 9966.168679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9966.183546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9967.276080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9967.289166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9967.329064] Lustre: lustre-MDT0000: new disk, initializing [ 9967.356952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9967.365463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9968.964344] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9973.424626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9973.481557] Lustre: 490418:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9973.518240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9973.522649] Lustre: Skipped 1 previous similar message [ 9973.576491] Lustre: lustre-MDT0001: new disk, initializing [ 9973.621453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9973.639000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9973.643644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9975.296149] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9979.046650] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9980.711852] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9983.736934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9983.925992] Lustre: lustre-OST0000: new disk, initializing [ 9983.929546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9983.934971] Lustre: 492158:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9983.992611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9986.774140] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 9986.776539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9986.782250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9986.839796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9990.623852] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9992.446313] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9992.580609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9994.554916] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9994.692331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9997.285211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9997.285818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9997.290205] LustreError: Skipped 1 previous similar message [ 9997.299628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9997.304618] Lustre: Skipped 1 previous similar message [10001.773937] Lustre: server umount lustre-OST0000 complete [10004.449926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10004.454971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10004.462764] Lustre: Skipped 1 previous similar message [10004.466447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10007.522862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10007.524379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10009.737845] Lustre: server umount lustre-MDT0000 complete [10011.530744] LustreError: 490409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780861721 with bad export cookie 1292636476127878336 [10011.536928] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10011.538144] LustreError: 490409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10011.720716] Lustre: server umount lustre-MDT0001 complete [10022.961453] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [10027.594594] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [10040.714134] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9681 sec ========== 15:49:11 (1780861751) [10041.553653] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:49:12 (1780861752) === [10042.994849] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:49:13 (1780861753) === [10060.692820] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [10062.239774] Key type lgssc unregistered [10062.414639] LNet: 498592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10062.419670] LNetError: 498592:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10062.428707] LNet: Removed LNI 192.168.202.104@tcp [10062.892329] Key type .llcrypt unregistered [10062.894463] Key type ._llcrypt unregistered