[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485594746 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2448MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.002557] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004079] kvm-guest: setup PV IPIs [ 0.007812] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009019] pid_max: default: 32768 minimum: 301 [ 0.010188] LSM: Security Framework initializing [ 0.011077] Yama: becoming mindful. [ 0.012081] SELinux: Initializing. [ 0.013106] *** VALIDATE selinux *** [ 0.024583] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030742] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032084] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033130] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035126] *** VALIDATE tmpfs *** [ 0.037285] *** VALIDATE proc *** [ 0.038389] *** VALIDATE cgroup *** [ 0.039011] *** VALIDATE cgroup2 *** [ 0.041169] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.043036] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.045008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.046042] Spectre V2 : User space: Vulnerable [ 0.047014] Speculative Store Bypass: Vulnerable [ 0.050825] debug: unmapping init [mem 0xffffffff9f259000-0xffffffff9f260fff] [ 0.053338] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.054939] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.056092] ... version: 2 [ 0.057011] ... bit width: 48 [ 0.058158] ... generic registers: 4 [ 0.059020] ... value mask: 0000ffffffffffff [ 0.060204] ... max period: 00007fffffffffff [ 0.061022] ... fixed-purpose events: 3 [ 0.062029] ... event mask: 000000070000000f [ 0.063401] rcu: Hierarchical SRCU implementation. [ 0.066087] smp: Bringing up secondary CPUs ... [ 0.067676] x86: Booting SMP configuration: [ 0.068035] .... node #0, CPUs: #1 #2 #3 [ 0.079045] smp: Brought up 1 node, 4 CPUs [ 0.081020] smpboot: Max logical packages: 1 [ 0.082020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.205035] node 0 deferred pages initialised in 121ms [ 0.209194] devtmpfs: initialized [ 0.210282] x86/mm: Memory block size: 128MB [ 0.215257] gcov: version magic: 0x41383552 [ 0.217651] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.218118] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.219519] pinctrl core: initialized pinctrl subsystem [ 0.220465] [ 0.221008] ************************************************************* [ 0.222011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.223016] ** ** [ 0.224032] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.225018] ** ** [ 0.226014] ** This means that this kernel is built to expose internal ** [ 0.227015] ** IOMMU data structures, which may compromise security on ** [ 0.228089] ** your system. ** [ 0.229017] ** ** [ 0.230013] ** If you see this message and you are not debugging the ** [ 0.231065] ** kernel, report this immediately to your vendor! ** [ 0.232025] ** ** [ 0.233038] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.234014] ************************************************************* [ 0.236230] NET: Registered protocol family 16 [ 0.237933] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.238128] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.239497] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.241241] cpuidle: using governor menu [ 0.243103] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.244881] PCI: Using configuration type 1 for base access [ 0.245137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.256143] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.258029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.261023] cryptd: max_cpu_qlen set to 1000 [ 0.263363] ACPI: Added _OSI(Module Device) [ 0.264023] ACPI: Added _OSI(Processor Device) [ 0.265018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.266018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.271058] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.275653] ACPI: Interpreter enabled [ 0.276065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.277015] ACPI: Using IOAPIC for interrupt routing [ 0.278109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.279525] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.292183] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.293045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.294084] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.295144] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.298033] acpiphp: Slot [2] registered [ 0.299163] acpiphp: Slot [5] registered [ 0.300129] acpiphp: Slot [6] registered [ 0.301636] acpiphp: Slot [7] registered [ 0.302268] acpiphp: Slot [8] registered [ 0.303238] acpiphp: Slot [9] registered [ 0.304291] acpiphp: Slot [10] registered [ 0.305304] acpiphp: Slot [3] registered [ 0.306129] acpiphp: Slot [4] registered [ 0.307122] acpiphp: Slot [11] registered [ 0.308101] acpiphp: Slot [12] registered [ 0.309107] acpiphp: Slot [13] registered [ 0.310097] acpiphp: Slot [14] registered [ 0.311068] acpiphp: Slot [15] registered [ 0.312096] acpiphp: Slot [16] registered [ 0.313127] acpiphp: Slot [17] registered [ 0.314222] acpiphp: Slot [18] registered [ 0.315195] acpiphp: Slot [19] registered [ 0.316125] acpiphp: Slot [20] registered [ 0.317251] acpiphp: Slot [21] registered [ 0.318250] acpiphp: Slot [22] registered [ 0.319143] acpiphp: Slot [23] registered [ 0.320100] acpiphp: Slot [24] registered [ 0.321256] acpiphp: Slot [25] registered [ 0.322688] acpiphp: Slot [26] registered [ 0.323178] acpiphp: Slot [27] registered [ 0.324211] acpiphp: Slot [28] registered [ 0.325167] acpiphp: Slot [29] registered [ 0.326096] acpiphp: Slot [30] registered [ 0.327232] acpiphp: Slot [31] registered [ 0.328085] PCI host bridge to bus 0000:00 [ 0.329036] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.330052] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.331042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.332036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333045] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.334057] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.335447] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.338787] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.341170] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.350590] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.351016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355895] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.357360] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.358039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.360416] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.363018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.371022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.374470] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.379691] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.382018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.385022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.392014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.398940] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.401019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.404015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.411017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.427339] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.456019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.471019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.489020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.503744] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.524021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.540018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.548000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.548000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.548000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.548000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.568017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.582547] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.593020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.602016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.610000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.610000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.610000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.610000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.610000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.610000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.610135] iommu: Default domain type: Passthrough [ 0.628009] SCSI subsystem initialized [ 0.629254] ACPI: bus type USB registered [ 0.631153] usbcore: registered new interface driver usbfs [ 0.634130] usbcore: registered new interface driver hub [ 0.636146] usbcore: registered new device driver usb [ 0.638173] pps_core: LinuxPPS API ver. 1 registered [ 0.640023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.644090] PTP clock support registered [ 0.649097] EDAC MC: Ver: 3.0.0 [ 0.663455] PCI: Using ACPI for IRQ routing [ 0.669030] NetLabel: Initializing [ 0.671012] NetLabel: domain hash size = 128 [ 0.674012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.676091] NetLabel: unlabeled traffic allowed by default [ 0.679351] vgaarb: loaded [ 0.681342] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.684016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.692786] clocksource: Switched to clocksource kvm-clock [ 0.821924] VFS: Disk quotas dquot_6.6.0 [ 0.826432] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.830985] *** VALIDATE ramfs *** [ 0.832537] *** VALIDATE hugetlbfs *** [ 0.834843] pnp: PnP ACPI init [ 0.839456] pnp: PnP ACPI: found 6 devices [ 0.860886] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.866264] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.869148] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.871145] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.874881] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.881802] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.886768] NET: Registered protocol family 2 [ 0.891292] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.900458] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.907911] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.917980] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.924089] TCP: Hash tables configured (established 65536 bind 65536) [ 0.933944] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.939177] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.943169] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.947905] NET: Registered protocol family 1 [ 0.951038] RPC: Registered named UNIX socket transport module. [ 0.955119] RPC: Registered udp transport module. [ 0.957292] RPC: Registered tcp transport module. [ 0.959392] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.962672] NET: Registered protocol family 44 [ 0.964537] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.967579] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.970245] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.974066] PCI: CLS 0 bytes, default 64 [ 0.976547] Unpacking initramfs... [ 3.026843] debug: unmapping init [mem 0xffff97f07cc54000-0xffff97f07ffbffff] [ 3.063262] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.069799] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 3.077721] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.894487] Initialise system trusted keyrings [ 3.896513] Key type blacklist registered [ 3.899208] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.909281] zbud: loaded [ 3.912917] *** VALIDATE nfs *** [ 3.914757] *** VALIDATE nfs4 *** [ 3.916498] pstore: using deflate compression [ 3.920746] Platform Keyring initialized [ 4.047887] NET: Registered protocol family 38 [ 4.050458] Key type asymmetric registered [ 4.052664] Asymmetric key parser 'x509' registered [ 4.056270] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.061047] io scheduler mq-deadline registered [ 4.063756] io scheduler kyber registered [ 4.066249] io scheduler bfq registered [ 4.068745] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.072991] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.076670] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.080843] ACPI: Power Button [PWRF] [ 4.087398] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.095854] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.128873] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.166639] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.196425] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.227803] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.258822] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.265369] Non-volatile memory driver v1.3 [ 4.267686] Linux agpgart interface v0.103 [ 4.314130] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 4.319233] vda: detected capacity change from 0 to 74645504 [ 4.440096] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.600365] vdb: detected capacity change from 0 to 1073741824 [ 4.680814] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.693871] vdc: detected capacity change from 0 to 2621440000 [ 4.751206] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.756144] vdd: detected capacity change from 0 to 2621440000 [ 4.771846] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.774359] vde: detected capacity change from 0 to 4294967296 [ 4.797653] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.800319] vdf: detected capacity change from 0 to 4294967296 [ 4.808962] libphy: Fixed MDIO Bus: probed [ 4.816274] usbcore: registered new interface driver usbserial_generic [ 4.819350] usbserial: USB Serial support registered for generic [ 4.822179] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.827032] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.829346] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.833681] mousedev: PS/2 mouse device common for all mice [ 4.840178] rtc_cmos 00:05: RTC can wake from S4 [ 4.849409] rtc_cmos 00:05: registered as rtc0 [ 4.850677] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.855120] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.855206] intel_pstate: CPU model not supported [ 4.862751] hid: raw HID events driver (C) Jiri Kosina [ 4.877408] usbcore: registered new interface driver usbhid [ 4.880296] usbhid: USB HID core driver [ 4.882811] drop_monitor: Initializing network drop monitor service [ 4.885861] Initializing XFRM netlink socket [ 4.889215] NET: Registered protocol family 10 [ 4.894521] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.897384] Segment Routing with IPv6 [ 4.904787] NET: Registered protocol family 17 [ 4.908386] mpls_gso: MPLS GSO support [ 4.912208] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.925916] RAS: Correctable Errors collector initialized. [ 4.928397] AVX version of gcm_enc/dec engaged. [ 4.930344] AES CTR mode by8 optimization enabled [ 5.102448] sched_clock: Marking stable (5102381669, 0)->(6938971611, -1836589942) [ 5.110899] registered taskstats version 1 [ 5.120262] Loading compiled-in X.509 certificates [ 5.123831] zswap: loaded using pool lzo/zbud [ 5.173695] Key type big_key registered [ 5.197964] Key type encrypted registered [ 5.201689] ima: No TPM chip found, activating TPM-bypass! [ 5.204982] ima: Allocated hash algorithm: sha1 [ 5.217922] ima: No architecture policies found [ 5.224738] evm: Initialising EVM extended attributes: [ 5.231465] evm: security.selinux [ 5.233714] evm: security.ima [ 5.235860] evm: security.capability [ 5.239687] evm: HMAC attrs: 0x1 [ 5.243924] rtc_cmos 00:05: setting system clock to 2026-07-16 20:11:29 UTC (1784232689) [ 5.260837] debug: unmapping init [mem 0xffffffffa0203000-0xffffffffa03fffff] [ 5.266657] debug: unmapping init [mem 0xffffffff9ef82000-0xffffffff9f258fff] [ 5.277800] Write protecting the kernel read-only data: 28672k [ 5.291792] debug: unmapping init [mem 0xffffffff9d603000-0xffffffff9d7fffff] [ 5.296439] debug: unmapping init [mem 0xffffffff9df14000-0xffffffff9dffffff] [ 5.373480] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.394878] systemd[1]: Detected virtualization kvm. [ 5.402222] systemd[1]: Detected architecture x86-64. [ 5.407950] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.445369] systemd[1]: No hostname configured. [ 5.447842] systemd[1]: Set hostname to . [ 5.450699] random: systemd: uninitialized urandom read (16 bytes read) [ 5.453823] systemd[1]: Initializing machine ID from random generator. [ 5.732229] random: ln: uninitialized urandom read (6 bytes read) [ 5.979446] random: systemd: uninitialized urandom read (16 bytes read) [ 5.983493] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.996197] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 6.001109] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.329120] device-mapper: uevent: version 1.0.3 [ 7.332643] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.074136] random: fast init done [ 9.076874] virtio_net virtio0 ens2: renamed from eth0 [ 9.550083] scsi host0: ata_piix [ 9.570222] scsi host1: ata_piix [ 9.572769] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.588542] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.753498] random: crng init done [ 14.760799] random: 7 urandom warning(s) missed due to ratelimiting [ 16.486486] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.173932] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.652462] printk: systemd: 22 output lines suppressed due to ratelimiting [ 21.431458] SELinux: Disabled at runtime. [ 21.552831] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.576135] systemd[1]: Detected virtualization kvm. [ 21.579104] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.239695] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.246687] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.268052] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.282479] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.289157] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.305162] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.317157] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 23.767133] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 24.776498] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.680408] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.765634] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.308463] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.405797] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.109791] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start j[ 32.080793] NFS: Registering the id_resolver key type ob is running for Configur…-on[ 32.093251] Key type id_resolver registered ly root support (9s / no limit)[ 32.111880] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg428-server login: [ 69.612923] libcfs: loading out-of-tree module taints kernel. [ 69.639794] Key type ._llcrypt registered [ 69.641558] Key type .llcrypt registered [ 69.705241] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 79.158035] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 79.903854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 79.912416] alg: No test for adler32 (adler32-zlib) [ 81.008619] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 81.370360] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 83.015172] Key type lgssc registered [ 83.792127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 92.444934] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 112.255865] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 118.373780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 118.390892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 119.524654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 119.541752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 119.614863] Lustre: lustre-MDT0000: new disk, initializing [ 119.663654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 119.677295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 121.644764] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 127.826940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 127.866414] Lustre: 6480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 127.881878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 127.885178] Lustre: Skipped 1 previous similar message [ 127.930494] Lustre: lustre-MDT0001: new disk, initializing [ 127.960284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 127.976334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 127.983144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 129.605329] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 132.174639] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 136.304075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 136.450241] Lustre: lustre-OST0000: new disk, initializing [ 136.452975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 136.459419] Lustre: 8419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 136.503347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.343909] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.351796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 143.357797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 143.401562] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 147.406388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 147.478209] Lustre: lustre-OST0001: new disk, initializing [ 147.480830] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 147.485391] Lustre: 9489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 147.519448] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.575174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 149.586599] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 149.629076] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 150.748817] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.964549] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 162.280769] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 165.534511] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing check_logdir /tmp/testlogs/ [ 167.948713] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing yml_node [ 170.068948] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 171.248233] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 172.494424] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 173.381497] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 16 16:14:17 EDT 2026 [ 182.120066] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 182.888889] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 195.551962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 195.552458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 195.557635] Lustre: Skipped 2 previous similar messages [ 195.561563] Lustre: Skipped 1 previous similar message [ 200.673336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 205.793056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 205.799971] Lustre: Skipped 6 previous similar messages [ 209.375123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 209.481231] Lustre: server umount lustre-MDT0000 complete [ 211.500852] LustreError: 10304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784232895 with bad export cookie 7409208205104663436 [ 211.502861] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 211.508367] LustreError: 10304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 216.031709] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 216.031760] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 216.032299] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 216.042072] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 216.057296] Lustre: Skipped 2 previous similar messages [ 217.793623] Lustre: server umount lustre-MDT0001 complete [ 219.932337] Lustre: server umount lustre-OST0000 complete [ 221.957133] Lustre: server umount lustre-OST0001 complete [ 225.245810] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 230.691441] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 249.285878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 249.443801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 249.463885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 249.525308] Lustre: lustre-MDT0000: new disk, initializing [ 249.592792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 249.619491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 251.688527] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 257.957783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.025689] Lustre: 15032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 258.061105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 258.064129] Lustre: Skipped 1 previous similar message [ 258.109687] Lustre: lustre-MDT0001: new disk, initializing [ 258.166282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 258.192697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 258.201536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 260.130841] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 265.152187] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 267.952832] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 271.750944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.961130] Lustre: 16999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 273.437155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 273.495699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 275.344550] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 281.025304] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 283.344890] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 283.471416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 285.504390] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 285.610856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 288.736217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 288.740056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.740070] LustreError: Skipped 1 previous similar message [ 288.749391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 288.755102] Lustre: Skipped 2 previous similar messages [ 292.789156] Lustre: server umount lustre-OST0000 complete [ 298.979237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.980755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 298.985517] Lustre: Skipped 2 previous similar messages [ 298.992582] Lustre: Skipped 1 previous similar message [ 309.215192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 309.319850] Lustre: server umount lustre-MDT0000 complete [ 311.487239] LustreError: 15827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784232995 with bad export cookie 7409208205104664955 [ 311.490520] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.493886] LustreError: 15827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 311.686760] Lustre: server umount lustre-MDT0001 complete [ 313.907212] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:16:37 (1784232997) [ 315.214334] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 316.117219] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:16:40 (1784233000) [ 319.935763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 320.206142] LustreError: 19408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.263976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 320.268527] Lustre: Skipped 1 previous similar message [ 322.615386] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 325.604104] LustreError: 19409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.587067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 330.308722] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 334.657859] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 337.985661] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 342.085120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 345.666714] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 349.549535] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 360.931452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 360.942128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.950894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 360.953424] Lustre: Skipped 4 previous similar messages [ 362.980232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 364.159765] Lustre: server umount lustre-OST0000 complete [ 372.704910] Lustre: server umount lustre-MDT0000 complete [ 373.216409] LustreError: 19404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.230353] LustreError: 19404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 375.117739] LustreError: 19390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233059 with bad export cookie 7409208205104665837 [ 375.118899] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.134512] LustreError: 19390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 375.346963] Lustre: server umount lustre-MDT0001 complete [ 382.017642] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 383.914518] Key type lgssc unregistered [ 384.148565] LNet: 22940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.160834] LNetError: 22940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 384.186338] LNet: Removed LNI 192.168.204.128@tcp [ 384.922605] Key type .llcrypt unregistered [ 384.924975] Key type ._llcrypt unregistered [ 399.495387] Key type ._llcrypt registered [ 399.497488] Key type .llcrypt registered [ 399.554617] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 400.288067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 400.305191] alg: No test for adler32 (adler32-zlib) [ 401.243994] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 401.394982] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 403.049974] Key type lgssc registered [ 403.779557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 408.747227] loop: module loaded [ 408.784271] loop0: detected capacity change from 0 to 204800000 [ 408.939595] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 408.948968] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 409.135291] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 412.415535] loop0: detected capacity change from 0 to 204800000 [ 412.541837] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 412.554910] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 412.564773] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 415.568118] loop0: detected capacity change from 0 to 204800000 [ 418.859446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 418.871810] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.047900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 420.091786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 420.159133] Lustre: lustre-MDT0000: new disk, initializing [ 420.204855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.222317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 422.782962] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 428.537802] loop1: detected capacity change from 0 to 204800000 [ 431.814589] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 431.984820] Lustre: lustre-OST07c6: new disk, initializing [ 431.994694] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 431.998355] Lustre: Skipped 1 previous similar message [ 432.003514] Lustre: 26101:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 432.053820] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 435.502328] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 437.319084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 437.333674] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 437.371915] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 439.605072] Lustre: 25090:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 439.609192] Lustre: 25090:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 439.612761] Lustre: 25090:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 439.617865] Lustre: 25090:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 439.622408] Lustre: 25090:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 439.626631] Lustre: 25090:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 454.508450] Lustre: 25089:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 454.515105] Lustre: 25089:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 454.518976] Lustre: 25089:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 454.522474] Lustre: 25089:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 454.528387] Lustre: 25089:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 454.534034] Lustre: 25089:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 458.712625] Lustre: 25089:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 458.717846] Lustre: 25089:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 458.721116] Lustre: 25089:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 458.724045] Lustre: 25089:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 458.728236] Lustre: 25089:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 458.731770] Lustre: 25089:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 458.735166] Lustre: 25089:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 458.739802] Lustre: 25089:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 458.743808] Lustre: 25089:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 458.748188] Lustre: 25089:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 458.752692] Lustre: 25089:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 458.757485] Lustre: 25089:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 461.861110] Lustre: 25091:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 461.866332] Lustre: 25091:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 461.870324] Lustre: 25091:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 461.874096] Lustre: 25091:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 461.877448] Lustre: 25091:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 461.880343] Lustre: 25091:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 461.883302] Lustre: 25091:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 461.887539] Lustre: 25091:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 461.891717] Lustre: 25091:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 461.894959] Lustre: 25091:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 461.897541] Lustre: 25091:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 461.900642] Lustre: 25091:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 468.788300] Lustre: 25089:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 468.796635] Lustre: 25089:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 468.803713] Lustre: 25089:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 468.808189] Lustre: 25089:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 468.812631] Lustre: 25089:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 468.818887] Lustre: 25089:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 468.823493] Lustre: 25089:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 468.828420] Lustre: 25089:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 468.832398] Lustre: 25089:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 468.837324] Lustre: 25089:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 468.843865] Lustre: 25089:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 468.849634] Lustre: 25089:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 478.816736] Lustre: 25089:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 478.822769] Lustre: 25089:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 478.825907] Lustre: 25089:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 478.830406] Lustre: 25089:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 478.836513] Lustre: 25089:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 478.841627] Lustre: 25089:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 478.849419] Lustre: 25089:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 478.855458] Lustre: 25089:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 478.858348] Lustre: 25089:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 478.861511] Lustre: 25089:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 478.864368] Lustre: 25089:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 478.868876] Lustre: 25089:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 523.029114] Lustre: server umount lustre-OST07c6 complete [ 538.690516] Lustre: server umount lustre-MDT0000 complete [ 547.912703] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 549.409532] Key type lgssc unregistered [ 549.582159] LNet: 28681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 549.589547] LNetError: 28681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 549.601444] LNet: Removed LNI 192.168.204.128@tcp [ 550.078860] Key type .llcrypt unregistered [ 550.080817] Key type ._llcrypt unregistered [ 561.423763] Key type ._llcrypt registered [ 561.425154] Key type .llcrypt registered [ 561.490908] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 561.999228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 562.039467] alg: No test for adler32 (adler32-zlib) [ 562.939694] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 563.070426] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 564.687282] Key type lgssc registered [ 565.351837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 569.727756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 569.739141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.027306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.002492] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 580.749460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.172558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 585.630947] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 593.306415] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 599.314942] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 607.751159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 608.100431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 613.970126] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 617.332790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 620.067790] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 636.902662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 636.905756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.912330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 636.923231] Lustre: Skipped 1 previous similar message [ 636.930478] Lustre: Skipped 1 previous similar message [ 642.044502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 642.049145] Lustre: Skipped 1 previous similar message [ 642.492522] Lustre: server umount lustre-OST0000 complete [ 647.141180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.144673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.151018] Lustre: Skipped 1 previous similar message [ 651.778660] Lustre: server umount lustre-MDT0000 complete [ 652.264086] LustreError: 29934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.281584] LustreError: 29934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 654.473175] LustreError: 29914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233338 with bad export cookie 3283578402823542875 [ 654.476133] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 654.480847] LustreError: 29914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 654.687923] Lustre: server umount lustre-MDT0001 complete [ 662.227093] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 664.214986] Key type lgssc unregistered [ 664.397577] LNet: 33669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.405192] LNetError: 33669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.415369] LNet: Removed LNI 192.168.204.128@tcp [ 664.931543] Key type .llcrypt unregistered [ 664.933477] Key type ._llcrypt unregistered [ 678.272477] Key type ._llcrypt registered [ 678.274466] Key type .llcrypt registered [ 678.355982] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 678.927233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 678.939168] alg: No test for adler32 (adler32-zlib) [ 679.921295] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 680.088237] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 681.703244] Key type lgssc registered [ 682.395307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 687.451841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 687.468426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 688.733094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.332301] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 696.779407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.015869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 699.619253] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 704.142502] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 707.204130] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 711.032692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 711.277637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 713.328118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 714.902477] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 719.102429] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 733.153385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 733.156148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.161851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 733.169930] Lustre: Skipped 1 previous similar message [ 733.176393] Lustre: Skipped 1 previous similar message [ 736.757988] Lustre: server umount lustre-OST0000 complete [ 743.393237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 743.394908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 743.410109] Lustre: Skipped 1 previous similar message [ 743.420927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 745.414208] Lustre: server umount lustre-MDT0000 complete [ 747.689136] LustreError: 34896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233431 with bad export cookie 9579699693902271774 [ 747.694739] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 747.701191] LustreError: 34896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 747.906125] Lustre: server umount lustre-MDT0001 complete [ 754.299503] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 756.246810] Key type lgssc unregistered [ 756.459803] LNet: 38468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.469090] LNetError: 38468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 756.487449] LNet: Removed LNI 192.168.204.128@tcp [ 757.031396] Key type .llcrypt unregistered [ 757.032850] Key type ._llcrypt unregistered [ 771.776225] Key type ._llcrypt registered [ 771.778640] Key type .llcrypt registered [ 771.840989] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 772.359756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 772.435178] alg: No test for adler32 (adler32-zlib) [ 773.425775] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 773.608740] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 775.215199] Key type lgssc registered [ 775.945652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 780.785549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 780.799123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 782.067669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 784.803904] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 790.007308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.237585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 792.686898] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 797.320262] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 800.345059] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 804.154241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 804.385078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.465958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 807.870789] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 811.959956] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 820.707376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.709971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 820.718392] Lustre: Skipped 1 previous similar message [ 820.727833] Lustre: Skipped 2 previous similar messages [ 823.842359] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 825.825225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 825.831809] Lustre: Skipped 2 previous similar messages [ 825.972156] Lustre: server umount lustre-MDT0000 complete [ 828.001298] LustreError: 39842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233512 with bad export cookie 7880032667536764807 [ 828.002162] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 828.010649] LustreError: 39842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 828.942595] LustreError: 39859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 828.952678] LustreError: 39859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 828.973629] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [ 830.945310] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.945844] LustreError: 39858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 830.951337] Lustre: Skipped 1 previous similar message [ 832.992771] LustreError: 41698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 832.992801] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 833.010419] Lustre: Skipped 1 previous similar message [ 834.276973] Lustre: server umount lustre-MDT0001 complete [ 861.151216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 861.239363] Lustre: server umount lustre-OST0000 complete [ 868.756215] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 870.150781] Key type lgssc unregistered [ 870.317798] LNet: 43717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.323536] LNetError: 43717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.337929] LNet: Removed LNI 192.168.204.128@tcp [ 870.798103] Key type .llcrypt unregistered [ 870.799604] Key type ._llcrypt unregistered [ 882.507668] Key type ._llcrypt registered [ 882.509600] Key type .llcrypt registered [ 882.567636] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 883.052335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 883.076114] alg: No test for adler32 (adler32-zlib) [ 883.995543] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 884.160280] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 885.775506] Key type lgssc registered [ 886.362474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 890.518620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 890.533121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 891.777533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 894.121211] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 898.815819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.037344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 901.119962] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 904.966616] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 907.507650] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 910.547945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 910.753179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.500622] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 916.608947] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 916.905768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 929.553396] Lustre: server umount lustre-OST0000 complete [ 944.608884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 944.609473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 944.614845] Lustre: Skipped 1 previous similar message [ 949.729443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 949.732153] Lustre: Skipped 2 previous similar messages [ 950.751295] Lustre: 44425:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784233618/real 1784233618] req@000000007123e72b x1870904497823232/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784233634 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 950.765316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 950.772279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 950.775419] Lustre: Skipped 1 previous similar message [ 952.831969] Lustre: server umount lustre-MDT0000 complete [ 954.377934] LustreError: 45092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233638 with bad export cookie 10303532403705911615 [ 954.383588] LustreError: 45092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 954.383659] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.592474] Lustre: server umount lustre-MDT0001 complete [ 965.105196] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 966.402623] Key type lgssc unregistered [ 966.557657] LNet: 48838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.562213] LNetError: 48838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.569548] LNet: Removed LNI 192.168.204.128@tcp [ 966.971157] Key type .llcrypt unregistered [ 966.973360] Key type ._llcrypt unregistered [ 977.966448] Key type ._llcrypt registered [ 977.967916] Key type .llcrypt registered [ 978.027753] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 978.411405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 978.424121] alg: No test for adler32 (adler32-zlib) [ 979.326409] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 979.436145] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 981.031145] Key type lgssc registered [ 981.571793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 985.313285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 985.324350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.600866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.560669] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 992.500595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.692729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 994.378525] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 997.553262] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 999.988415] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1002.819126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1002.979726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.446422] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1008.571420] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1012.201425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1021.370616] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1022.777876] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1033.677128] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-1616h28m37s [ 1034.312344] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1035.800976] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 1036.811797] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 1037.794827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.801966] Lustre: Skipped 2 previous similar messages [ 1038.819202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1038.822928] Lustre: Skipped 5 previous similar messages [ 1041.310385] Lustre: server umount lustre-MDT0000 complete [ 1042.708672] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1042.955219] LustreError: 50230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.963801] LustreError: 50230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1044.960820] LustreError: 50234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1044.967663] LustreError: 50234:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1045.453879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.503861] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1045.613938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.639508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1046.628181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1047.173519] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1048.566728] Lustre: DEBUG MARKER: Wait for df (36568) ... [ 1049.199181] Lustre: DEBUG MARKER: done [ 1051.104083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1051.108600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.115768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1052.130638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.387499] Lustre: server umount lustre-OST0000 complete [ 1056.178246] Lustre: server umount lustre-MDT0000 complete [ 1057.248931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.249713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1057.256659] Lustre: Skipped 1 previous similar message [ 1057.258410] LustreError: 52070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1057.275038] LustreError: 52070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1057.811422] LustreError: 50214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233742 with bad export cookie 5205574043256625784 [ 1057.817922] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1059.974859] Lustre: server umount lustre-MDT0001 complete [ 1064.442819] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1065.755804] Key type lgssc unregistered [ 1065.903902] LNet: 55166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.907428] LNetError: 55166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.914706] LNet: Removed LNI 192.168.204.128@tcp [ 1066.327942] Key type .llcrypt unregistered [ 1066.330075] Key type ._llcrypt unregistered [ 1083.264429] Key type ._llcrypt registered [ 1083.266053] Key type .llcrypt registered [ 1083.318555] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1083.827725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1083.840091] alg: No test for adler32 (adler32-zlib) [ 1084.727739] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 1084.839020] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1086.431172] Key type lgssc registered [ 1086.927530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1090.038779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1090.048336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1091.159507] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1091.176698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1091.251698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.872669] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1095.973160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.018470] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1096.032851] Lustre: 57355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1096.148265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1097.755256] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1100.735451] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1103.009756] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1105.845292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1105.933817] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1106.022909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.348911] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1109.456276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1111.102957] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1114.684586] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-1616h29m58s [ 1115.337766] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1118.187478] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1119.242101] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1119.260688] LustreError: 57364:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1139.738325] Lustre: lustre-MDT0000: Client 261f501d-9fa8-4d17-a7e6-60c2b58494e9 (at 192.168.204.28@tcp) reconnecting [ 1159.303163] LustreError: 57364:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1160.062611] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1160.938494] Lustre: DEBUG MARKER: done [ 1164.768847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1164.772516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.777654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1165.793067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1165.794203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1165.799792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.658067] Lustre: server umount lustre-OST0000 complete [ 1170.913086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.914062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1170.917633] Lustre: Skipped 1 previous similar message [ 1170.919813] Lustre: Skipped 1 previous similar message [ 1176.161938] Lustre: server umount lustre-MDT0000 complete [ 1177.398246] LustreError: 57349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233861 with bad export cookie 14581498306253491134 [ 1177.400800] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1177.402600] LustreError: 57349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1177.519975] Lustre: server umount lustre-MDT0001 complete [ 1181.313781] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1182.424808] Key type lgssc unregistered [ 1182.550396] LNet: 61678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.552941] LNetError: 61678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.564287] LNet: Removed LNI 192.168.204.128@tcp [ 1182.848638] Key type .llcrypt unregistered [ 1182.850459] Key type ._llcrypt unregistered [ 1198.444740] Key type ._llcrypt registered [ 1198.445715] Key type .llcrypt registered [ 1198.479472] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1198.938080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.947508] alg: No test for adler32 (adler32-zlib) [ 1199.798408] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 1199.879922] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1201.463170] Key type lgssc registered [ 1201.818765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1202.901459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1202.907780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.994921] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1204.003959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1204.058215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1204.392340] Lustre: Failing over lustre-MDT0000 [ 1204.482246] Lustre: server umount lustre-MDT0000 complete [ 1206.462221] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:31:30 (1784233890) [ 1208.468621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.597280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.826245] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1212.459968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1212.491229] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1212.501343] Lustre: 64453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1212.590898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1213.834174] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1216.220231] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1218.074960] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1220.290789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1220.359974] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1220.426929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1222.311700] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1223.951058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1224.669523] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1228.761902] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1233.375882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1233.379130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.384261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1234.400664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1234.403862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1234.407281] Lustre: Skipped 1 previous similar message [ 1236.823142] Lustre: server umount lustre-MDT0000 complete [ 1238.009131] LustreError: 64484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233922 with bad export cookie 15430649287131565087 [ 1238.012424] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1238.014693] LustreError: 64484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1238.103433] Lustre: server umount lustre-MDT0001 complete [ 1239.186475] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1240.721677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1243.384158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.671652] LustreError: 63375:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000052d55e77 x1870904829180032/t0(0) o250->MGC192.168.204.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1248.800987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.826080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1250.058785] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1252.878530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.974445] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.056165] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1254.377096] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1255.071163] Lustre: 63376:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784233923/real 1784233923] req@00000000ce1e6249 x1870904829179648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784233939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1256.713182] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1258.512901] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1259.487172] Lustre: 63377:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784233927/real 1784233927] req@0000000020aa3d81 x1870904829179776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784233943 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1261.215175] Lustre: 63378:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784233928/real 1784233928] req@0000000000784202 x1870904829179904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784233944 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1263.584152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1263.587585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.593842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.595560] Lustre: Skipped 2 previous similar messages [ 1264.607181] Lustre: 63379:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784233932/real 1784233932] req@00000000db9864e6 x1870904829180160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784233948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1266.331458] Lustre: server umount lustre-MDT0000 complete [ 1267.617966] LustreError: 64443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233951 with bad export cookie 15430649287131567355 [ 1267.621575] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1267.622930] LustreError: 64443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1267.724243] Lustre: server umount lustre-MDT0001 complete [ 1270.033927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1273.029394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.831848] LustreError: 63375:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000031688e7c x1870904829200896/t0(0) o250->MGC192.168.204.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1276.946777] LustreError: 66301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.953617] LustreError: 66301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1276.969786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1276.972130] Lustre: Skipped 1 previous similar message [ 1277.000682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1278.326145] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1281.149453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.246856] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.251682] Lustre: Skipped 2 previous similar messages [ 1282.663484] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1284.392371] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1284.394099] Lustre: Skipped 1 previous similar message [ 1285.076136] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1285.407088] Lustre: 63378:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784233953/real 1784233953] req@00000000e6b50734 x1870904829200512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784233969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1286.860592] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1289.697332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.698701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1289.701672] Lustre: Skipped 2 previous similar messages [ 1289.703831] Lustre: Skipped 4 previous similar messages [ 1294.817731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.820433] Lustre: Skipped 1 previous similar message [ 1294.942375] Lustre: server umount lustre-MDT0000 complete [ 1296.196040] LustreError: 64444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784233980 with bad export cookie 15430649287131568734 [ 1296.199181] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1296.199946] LustreError: 64444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1296.305327] Lustre: server umount lustre-MDT0001 complete [ 1298.573504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1299.691372] Lustre: DEBUG MARKER: files compared the same [ 1310.623279] Lustre: server umount lustre-OST0000 complete [ 1316.056256] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1317.044662] Key type lgssc unregistered [ 1317.167425] LNet: 74170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1317.169909] LNetError: 74170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.177257] LNet: Removed LNI 192.168.204.128@tcp [ 1317.471031] Key type .llcrypt unregistered [ 1317.472128] Key type ._llcrypt unregistered [ 1325.987743] Key type ._llcrypt registered [ 1325.988866] Key type .llcrypt registered [ 1326.025429] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1326.342667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1326.358399] alg: No test for adler32 (adler32-zlib) [ 1327.207218] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 1327.296868] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1328.879144] Key type lgssc registered [ 1329.252138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1331.734754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1331.742719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1332.899969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.281780] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug +malloc all [ 1337.218958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.337087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1338.705603] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug +malloc all [ 1340.938801] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1342.644771] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1344.804841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1344.897400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.662142] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug +malloc all [ 1348.970971] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1358.242154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1362.399511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1362.401831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.405657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1363.423774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1363.423891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1363.423954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1367.823500] Lustre: server umount lustre-OST0000 complete [ 1373.151576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1373.155393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1373.161075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1373.163192] Lustre: Skipped 1 previous similar message [ 1378.783525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1378.786927] Lustre: Skipped 2 previous similar messages [ 1383.391117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1383.446097] Lustre: server umount lustre-MDT0000 complete [ 1383.903545] LustreError: 75563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.907985] LustreError: 75563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1384.650861] LustreError: 75543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784234068 with bad export cookie 11637686544582811212 [ 1384.652200] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1384.655480] LustreError: 75543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1384.759370] Lustre: server umount lustre-MDT0001 complete [ 1388.433860] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1389.501447] Key type lgssc unregistered [ 1389.631333] LNet: 79113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.634155] LNetError: 79113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.642462] LNet: Removed LNI 192.168.204.128@tcp [ 1389.952053] Key type .llcrypt unregistered [ 1389.953240] Key type ._llcrypt unregistered [ 1398.403936] Key type ._llcrypt registered [ 1398.405285] Key type .llcrypt registered [ 1398.437727] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1398.783626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1398.791299] alg: No test for adler32 (adler32-zlib) [ 1399.650397] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 1399.745837] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1401.327142] Key type lgssc registered [ 1401.718375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1404.323909] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1404.329464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1419.743409] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1424.863256] LustreError: 80479:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 1451.487288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.535896] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1455.929251] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1458.468699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.507444] Lustre: *** cfs_fail_loc=706, val=0*** [ 1458.644269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.666716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1459.980047] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1462.700041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1462.823963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1464.119896] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1466.579570] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1468.381093] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1470.946233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.947679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1470.949653] Lustre: Skipped 1 previous similar message [ 1470.952611] Lustre: Skipped 1 previous similar message [ 1475.144047] Lustre: server umount lustre-OST0000 complete [ 1478.111867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1478.114901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.119627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.184071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.184601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.530426] Lustre: server umount lustre-MDT0000 complete [ 1483.733175] LustreError: 80485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784234167 with bad export cookie 11376744191523040175 [ 1483.736652] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1483.737319] LustreError: 80485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1483.848699] Lustre: server umount lustre-MDT0001 complete [ 1487.472446] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1488.426929] Key type lgssc unregistered [ 1488.553358] LNet: 84089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.555528] LNetError: 84089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1488.564251] LNet: Removed LNI 192.168.204.128@tcp [ 1488.844325] Key type .llcrypt unregistered [ 1488.845405] Key type ._llcrypt unregistered [ 1497.818192] Key type ._llcrypt registered [ 1497.819943] Key type .llcrypt registered [ 1497.855263] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1498.110846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1498.148656] alg: No test for adler32 (adler32-zlib) [ 1499.005950] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 1499.106098] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1500.687166] Key type lgssc registered [ 1501.048332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1503.583592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1503.591337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1505.929296] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1508.416773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.586345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1510.004243] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1512.817593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1512.945170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.814032] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1517.641175] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1517.646298] Lustre: MGS: Client ad6ea135-c9e3-4e91-b119-3194fceaad3f (at 0@lo) reconnecting [ 1517.758855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.809409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1519.159637] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1528.288300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1528.292135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1528.298765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1529.315309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1529.315780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1529.318102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1532.943124] Lustre: server umount lustre-OST0000 complete [ 1534.432473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1534.432584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.433435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1534.433439] Lustre: Skipped 1 previous similar message [ 1534.443484] Lustre: Skipped 1 previous similar message [ 1539.552211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1539.554346] Lustre: Skipped 2 previous similar messages [ 1540.295790] Lustre: server umount lustre-MDT0000 complete [ 1541.692671] Lustre: server umount lustre-MDT0001 complete [ 1542.986120] Lustre: server umount MGS complete [ 1546.692540] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:37:10 (1784234230) [ 1556.391859] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 1559.670793] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1571.509201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.956789] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1576.598706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1576.677823] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1576.696055] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1576.730202] Lustre: lustre-MDT0001: new disk, initializing [ 1576.750979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1576.759150] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1576.761625] Lustre: 93331:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1576.765284] LustreError: 93331:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1577.939889] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1581.877471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1581.985669] Lustre: lustre-OST0000: new disk, initializing [ 1581.988856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1581.993262] Lustre: 94095:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1582.017241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1583.968924] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1587.655050] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1589.844363] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1589.849169] Lustre: MGS: Client d04766b0-7f4e-4bc0-987e-9c5b0735d89f (at 0@lo) reconnecting [ 1589.864813] Lustre: 92601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1589.872176] LustreError: 93315:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1589.875040] LustreError: 93315:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1589.881395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1589.919596] Lustre: lustre-MDT0000: new disk, initializing [ 1589.970184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1590.751827] Lustre: 95116:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000005884dd43 x1870905142711936/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784234291 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1591.295680] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1592.799764] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1595.810264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1595.815114] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1596.131598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1615.843412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.844078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1615.847422] Lustre: Skipped 1 previous similar message [ 1621.075313] Lustre: server umount lustre-OST0000 complete [ 1626.080853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1626.081212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1626.085643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1626.089123] Lustre: Skipped 1 previous similar message [ 1626.092251] Lustre: Skipped 4 previous similar messages [ 1628.489745] Lustre: server umount lustre-MDT0000 complete [ 1635.942295] Lustre: server umount lustre-MDT0001 complete [ 1637.051308] Lustre: server umount MGS complete [ 1640.673439] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:38:44 (1784234324) [ 1646.350961] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1647.323795] Key type lgssc unregistered [ 1647.448402] LNet: 97661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1647.450909] LNetError: 97661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1647.458290] LNet: Removed LNI 192.168.204.128@tcp [ 1647.744509] Key type .llcrypt unregistered [ 1647.745750] Key type ._llcrypt unregistered [ 1654.939532] Key type ._llcrypt registered [ 1654.940727] Key type .llcrypt registered [ 1654.979189] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1655.415992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1655.431790] alg: No test for adler32 (adler32-zlib) [ 1656.308271] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 1656.412905] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1657.991154] Key type lgssc registered [ 1658.422182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1661.142351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1661.148157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.323908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1663.841326] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1666.933317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.081498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1668.477898] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1670.998873] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1672.919886] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1675.143863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1675.261309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.140802] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1679.485378] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1688.614138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1693.666304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.669528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1693.670162] Lustre: Skipped 1 previous similar message [ 1693.673736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1694.732513] Lustre: server umount lustre-OST0000 complete [ 1697.759909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1697.762089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.766422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1697.769334] Lustre: Skipped 1 previous similar message [ 1697.998598] Lustre: server umount lustre-MDT0000 complete [ 1698.784455] LustreError: 99599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1698.784648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.791287] LustreError: 99599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1699.175283] LustreError: 98839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784234383 with bad export cookie 11849108264611067969 [ 1699.177956] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1699.179908] LustreError: 98839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1699.281943] Lustre: server umount lustre-MDT0001 complete [ 1702.525867] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1703.392957] Key type lgssc unregistered [ 1703.509454] LNet: 102409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.513769] LNetError: 102409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.523329] LNet: Removed LNI 192.168.204.128@tcp [ 1703.805053] Key type .llcrypt unregistered [ 1703.806242] Key type ._llcrypt unregistered [ 1710.224547] Key type ._llcrypt registered [ 1710.225670] Key type .llcrypt registered [ 1710.258325] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1710.658572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1710.666252] alg: No test for adler32 (adler32-zlib) [ 1711.520192] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 1711.609568] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1713.191155] Key type lgssc registered [ 1713.559475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1715.527738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1715.530193] LustreError: 103468:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1715.531593] LustreError: 103470:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1715.536089] LustreError: 103468:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1715.538599] LustreError: 103468:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1715.542230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.542930] LustreError: 103468:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1716.702494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1719.273888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.375681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1720.926545] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1722.606287] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing lsmod [ 1723.586395] LustreError: 104673:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1723.586497] LustreError: 104674:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1723.590985] LustreError: 104673:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 1723.593514] LustreError: 104673:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1723.596641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1723.598502] LustreError: 104673:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1723.723602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.992996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1728.994312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1728.996709] Lustre: Skipped 2 previous similar messages [ 1728.999844] Lustre: Skipped 2 previous similar messages [ 1730.017632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1730.019947] Lustre: Skipped 2 previous similar messages [ 1730.775870] Lustre: server umount lustre-MDT0000 complete [ 1731.991323] LustreError: 103483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784234416 with bad export cookie 9847006585887709785 [ 1731.994823] LustreError: 103483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1731.995315] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1732.052200] LustreError: 103938:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000023103562 x1870905364987776/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1732.107698] Lustre: server umount lustre-MDT0001 complete [ 1743.385472] Lustre: server umount lustre-OST0000 complete [ 1746.221720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.388874] LustreError: 105738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1746.398175] LustreError: 105738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1746.415364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1747.779180] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1750.676797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1750.811259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1752.244367] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1754.794959] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1756.710564] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1758.800092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1758.894685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.540136] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1762.763697] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1771.174199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1776.610021] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1776.610657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1776.611794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1776.617876] Lustre: Skipped 1 previous similar message [ 1780.368903] Lustre: server umount lustre-OST0000 complete [ 1781.728620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1781.728754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.729458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1781.729464] Lustre: Skipped 1 previous similar message [ 1781.731400] LustreError: Skipped 1 previous similar message [ 1781.739741] Lustre: Skipped 1 previous similar message [ 1786.848696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.850993] Lustre: Skipped 2 previous similar messages [ 1787.801152] Lustre: server umount lustre-MDT0000 complete [ 1788.874304] LustreError: 105720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784234473 with bad export cookie 9847006585887710471 [ 1788.876701] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1788.877531] LustreError: 105720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1788.980887] Lustre: server umount lustre-MDT0001 complete [ 1792.354432] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1793.299350] Key type lgssc unregistered [ 1793.422358] LNet: 109288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.425562] LNetError: 109288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.433287] LNet: Removed LNI 192.168.204.128@tcp [ 1793.694570] Key type .llcrypt unregistered [ 1793.696226] Key type ._llcrypt unregistered [ 1802.429867] Key type ._llcrypt registered [ 1802.431878] Key type .llcrypt registered [ 1802.474224] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1802.892574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1802.910402] alg: No test for adler32 (adler32-zlib) [ 1803.779378] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 1803.878654] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1805.463149] Key type lgssc registered [ 1805.837257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1808.539639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1808.546040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1809.699785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1811.082431] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1813.897595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.029779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1815.317927] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1817.613313] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1819.433449] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1821.651160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1821.768616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.740284] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1826.157776] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1835.110578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1838.561950] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1839.361395] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1840.612045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1840.612782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.619475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1840.621322] Lustre: Skipped 1 previous similar message [ 1845.729281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1846.421194] Lustre: server umount lustre-OST0000 complete [ 1849.823859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1849.826536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.832662] Lustre: Skipped 1 previous similar message [ 1849.835073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1849.837525] Lustre: Skipped 1 previous similar message [ 1850.849035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.852418] Lustre: server umount lustre-MDT0000 complete [ 1855.188510] LustreError: 113227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784234539 with bad export cookie 11110598010050518731 [ 1855.191315] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1855.192788] LustreError: 113227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1855.326972] Lustre: server umount lustre-MDT0001 complete [ 1858.979264] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1860.053586] Key type lgssc unregistered [ 1860.194766] LNet: 114330:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1860.198487] LNetError: 114330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1860.206423] LNet: Removed LNI 192.168.204.128@tcp [ 1860.566174] Key type .llcrypt unregistered [ 1860.567830] Key type ._llcrypt unregistered [ 1867.812377] Key type ._llcrypt registered [ 1867.813918] Key type .llcrypt registered [ 1867.846607] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1868.267277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1868.279374] alg: No test for adler32 (adler32-zlib) [ 1869.130638] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 1869.221792] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1870.799203] Key type lgssc registered [ 1871.192695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1873.592932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1873.600142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1874.742849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1875.971709] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1878.625837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.744591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1879.989350] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1882.243833] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1884.026062] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1886.136932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1886.248806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.926450] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1890.114862] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1899.558297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1903.584065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1903.587511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.593173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1904.610520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1904.611714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1904.612400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1904.618513] Lustre: Skipped 1 previous similar message [ 1905.236416] Lustre: server umount lustre-OST0000 complete [ 1908.631692] Lustre: server umount lustre-MDT0000 complete [ 1909.728519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1909.728744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.735252] LustreError: 116284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1909.739871] LustreError: 116284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1909.812256] LustreError: 115525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784234594 with bad export cookie 11231156285060293946 [ 1909.815902] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1909.816147] LustreError: 115525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1909.933819] Lustre: server umount lustre-MDT0001 complete [ 1913.391181] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1914.380323] Key type lgssc unregistered [ 1914.494488] LNet: 119094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.498302] LNetError: 119094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.507285] LNet: Removed LNI 192.168.204.128@tcp [ 1914.819512] Key type .llcrypt unregistered [ 1914.821292] Key type ._llcrypt unregistered [ 1923.962155] Key type ._llcrypt registered [ 1923.963853] Key type .llcrypt registered [ 1924.007588] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1924.321475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1924.331041] alg: No test for adler32 (adler32-zlib) [ 1925.193388] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 1925.286514] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1926.863130] Key type lgssc registered [ 1927.232346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1929.924200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1929.930547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.080126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.406379] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1935.156628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.297128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1936.587930] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1938.952250] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1940.800382] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1943.018365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1943.131465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.149674] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1947.589868] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1956.454478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1960.336957] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1961.540427] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1961.543311] Lustre: Skipped 2 previous similar messages [ 1962.981308] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1962.984925] Lustre: Skipped 2 previous similar messages [ 1968.124577] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1968.136028] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1970.564886] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.28@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1970.568775] Lustre: Skipped 2 previous similar messages [ 1975.805462] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.28@tcp 192.168.0.[2,10]@tcp [ 1977.113018] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.28/24@tcp in log lustre-MDT0000 [ 1977.116930] Lustre: Skipped 2 previous similar messages [ 1981.948249] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.28/24@tcp [ 1981.951301] Lustre: Skipped 1 previous similar message [ 1986.528331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1986.530586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.534322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1987.552898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1987.553812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.554465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1991.820318] Lustre: server umount lustre-OST0000 complete [ 1996.767874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1996.770394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.775755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1996.777937] Lustre: Skipped 1 previous similar message [ 1999.192836] Lustre: server umount lustre-MDT0000 complete [ 2000.305323] LustreError: 120469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784234684 with bad export cookie 12826003709056851071 [ 2000.307540] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2000.308506] LustreError: 120469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2000.409492] Lustre: server umount lustre-MDT0001 complete [ 2003.649168] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2004.567118] Key type lgssc unregistered [ 2004.692279] LNet: 125631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2004.694573] LNetError: 125631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2004.700270] LNet: Removed LNI 192.168.204.128@tcp [ 2004.984131] Key type .llcrypt unregistered [ 2004.985553] Key type ._llcrypt unregistered [ 2014.420902] Key type ._llcrypt registered [ 2014.422174] Key type .llcrypt registered [ 2014.459075] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2014.764520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2014.811546] alg: No test for adler32 (adler32-zlib) [ 2015.656092] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 2015.739667] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2017.311148] Key type lgssc registered [ 2017.625951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2020.140841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2020.147466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.297666] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2021.299771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.306312] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.28/24@tcp [ 2022.654103] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2025.512101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.622668] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2025.630288] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.28/24@tcp [ 2025.645038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2027.002180] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2029.466768] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2031.264224] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2033.555174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2033.675742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.595655] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2038.084237] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2045.988290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2056.162306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.163068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2056.164209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2056.167050] Lustre: Skipped 1 previous similar message [ 2056.171121] LustreError: Skipped 1 previous similar message [ 2057.615167] Lustre: server umount lustre-OST0000 complete [ 2061.281403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.282312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2061.285012] Lustre: Skipped 1 previous similar message [ 2061.289238] Lustre: Skipped 2 previous similar messages [ 2065.035411] Lustre: server umount lustre-MDT0000 complete [ 2066.164142] LustreError: 127151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784234750 with bad export cookie 593311139676600846 [ 2066.167150] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2066.168747] LustreError: 127151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2066.282453] Lustre: server umount lustre-MDT0001 complete [ 2069.596278] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2070.532524] Key type lgssc unregistered [ 2070.648370] LNet: 130774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.650582] LNetError: 130774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.657293] LNet: Removed LNI 192.168.204.128@tcp [ 2070.937868] Key type .llcrypt unregistered [ 2070.938880] Key type ._llcrypt unregistered [ 2078.769675] Key type ._llcrypt registered [ 2078.770969] Key type .llcrypt registered [ 2078.804475] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 2084.296314] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2084.580179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2084.620035] alg: No test for adler32 (adler32-zlib) [ 2085.457533] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 2085.531215] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2087.111176] Key type lgssc registered [ 2087.407193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2097.427480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2097.433136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2098.526013] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2098.535854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2098.569977] Lustre: lustre-MDT0000: new disk, initializing [ 2098.589278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.594631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2099.951770] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2103.891715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.929818] Lustre: 134144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2103.951693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2103.953884] Lustre: Skipped 1 previous similar message [ 2103.987971] Lustre: lustre-MDT0001: new disk, initializing [ 2104.018852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2104.030614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2104.035204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2105.253472] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2108.759847] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2110.477130] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2112.656131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2112.767310] Lustre: lustre-OST0000: new disk, initializing [ 2112.770221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2112.773468] Lustre: 136113:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2112.808903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.363260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2114.366939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2114.402871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2114.759323] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2118.361760] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2119.649497] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2119.716328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2120.992236] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2121.063466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2123.722674] Lustre: server umount lustre-OST0000 complete [ 2129.889854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.890134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2129.890751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2129.894807] Lustre: Skipped 1 previous similar message [ 2135.008047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2135.011029] Lustre: Skipped 2 previous similar messages [ 2135.031988] Lustre: server umount lustre-MDT0000 complete [ 2136.164910] LustreError: 134942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784234820 with bad export cookie 9551342206308013687 [ 2136.168366] LustreError: 134942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2136.168447] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2145.808545] Lustre: server umount lustre-MDT0001 complete [ 2148.400277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.568186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.759502] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2152.490017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.609458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2153.873145] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2156.127537] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2157.836083] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2159.933018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2161.819622] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2164.093596] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2165.311589] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2174.522860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2175.771581] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2175.835824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2178.997961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2179.050395] Lustre: lustre-OST0001: new disk, initializing [ 2179.052596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2179.055955] Lustre: 141763:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2179.083483] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2179.086453] Lustre: Skipped 1 previous similar message [ 2180.889550] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2183.971808] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2184.693850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2184.697407] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2184.720268] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2185.054915] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2186.285871] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2186.348406] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2187.906231] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d3dd8116000.ost_server_uuid 50 [ 2188.334423] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d3dd8116000.ost_server_uuid in FULL state after 0 sec [ 2194.914803] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2194.915329] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.917285] LustreError: Skipped 1 previous similar message [ 2194.921365] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2194.924573] Lustre: Skipped 1 previous similar message [ 2196.090756] Lustre: server umount lustre-OST0001 complete [ 2197.983854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2197.988233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2197.995125] Lustre: Skipped 1 previous similar message [ 2197.998082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2200.033573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2200.035289] LustreError: 140147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.035535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.045437] LustreError: 140147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2203.338841] Lustre: server umount lustre-OST0000 complete [ 2205.153044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.153896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2205.158035] Lustre: Skipped 1 previous similar message [ 2205.162975] Lustre: Skipped 3 previous similar messages [ 2210.708888] Lustre: server umount lustre-MDT0000 complete [ 2211.894165] LustreError: 138309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784234896 with bad export cookie 9551342206308014555 [ 2211.898391] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2211.899553] LustreError: 138309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2212.030260] Lustre: server umount lustre-MDT0001 complete [ 2218.729921] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2219.893515] Key type lgssc unregistered [ 2220.026482] LNet: 144767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.030215] LNetError: 144767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.043367] LNet: Removed LNI 192.168.204.128@tcp [ 2220.375687] Key type .llcrypt unregistered [ 2220.377610] Key type ._llcrypt unregistered [ 2233.365303] Key type ._llcrypt registered [ 2233.367083] Key type .llcrypt registered [ 2233.406408] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 2239.305238] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2239.647761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2239.692876] alg: No test for adler32 (adler32-zlib) [ 2240.555584] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 2240.638674] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2242.215147] Key type lgssc registered [ 2242.559493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2253.465964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2253.471856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.556714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2254.568094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2254.617290] Lustre: lustre-MDT0000: new disk, initializing [ 2254.639148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.644745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2256.003705] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2260.151648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.194566] Lustre: 148890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2260.220693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2260.223136] Lustre: Skipped 1 previous similar message [ 2260.260616] Lustre: lustre-MDT0001: new disk, initializing [ 2260.291568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2260.305101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2260.308139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2261.715383] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2265.538226] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2267.455119] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2269.761660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2269.875717] Lustre: lustre-OST0000: new disk, initializing [ 2269.877345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2269.879977] Lustre: 150859:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2269.913481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.910547] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2275.637959] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2276.858154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2276.862812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2276.896823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2282.805274] Lustre: Failing over lustre-OST0000 [ 2282.860280] Lustre: server umount lustre-OST0000 complete [ 2295.731692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2295.847659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.854601] Lustre: Skipped 1 previous similar message [ 2295.855212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.863858] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2297.362682] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2297.854662] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2299.149472] Lustre: Failing over lustre-MDT0000 [ 2299.238615] Lustre: server umount lustre-MDT0000 complete [ 2300.897910] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.901458] Lustre: Skipped 2 previous similar messages [ 2302.943134] Lustre: 147122:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784234971/real 1784234971] req@000000006735123e x1870905919697152/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784234987 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2306.016091] LustreError: 148901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.020214] LustreError: 148901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2307.595200] LustreError: 148896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.063134] Lustre: 147119:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784234976/real 1784234976] req@0000000021c89cad x1870905919698816/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784234992 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2308.069250] Lustre: 147119:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2311.135943] LustreError: 148902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2311.140347] LustreError: 148902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2311.880389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2311.917468] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2311.921309] Lustre: 148894:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.128@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2311.988274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2312.004833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.034173] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2312.714950] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2313.253210] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2317.285432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2317.294116] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2317.321013] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2322.402350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2322.405750] LustreError: Skipped 1 previous similar message [ 2322.405775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2322.408266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2322.416334] Lustre: Skipped 1 previous similar message [ 2322.421033] Lustre: Skipped 1 previous similar message [ 2324.381567] Lustre: server umount lustre-OST0000 complete [ 2327.520486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2327.520798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2327.528133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2327.530626] Lustre: Skipped 1 previous similar message [ 2332.640794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2332.643536] Lustre: Skipped 1 previous similar message [ 2337.759944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.762591] Lustre: Skipped 1 previous similar message [ 2340.319896] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2340.408527] Lustre: server umount lustre-MDT0000 complete [ 2341.856886] LustreError: 148902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.864582] LustreError: 148902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2341.913237] LustreError: 149684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784235026 with bad export cookie 8691484182232991622 [ 2341.916591] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2341.921684] LustreError: 149684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2342.062338] Lustre: server umount lustre-MDT0001 complete [ 2345.756751] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2346.744609] Key type lgssc unregistered [ 2346.869372] LNet: 154509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2346.872323] LNetError: 154509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2346.881294] LNet: Removed LNI 192.168.204.128@tcp [ 2347.183374] Key type .llcrypt unregistered [ 2347.184501] Key type ._llcrypt unregistered [ 2366.505696] Key type ._llcrypt registered [ 2366.506623] Key type .llcrypt registered [ 2366.537231] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2366.928857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.944321] alg: No test for adler32 (adler32-zlib) [ 2367.778570] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 2367.850616] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2369.431096] Key type lgssc registered [ 2369.814133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2372.478237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2372.485586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.624111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2374.913694] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2378.487933] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2380.476853] LustreError: 156056:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2380.480296] LustreError: 156056:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2380.482934] LustreError: 156056:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2380.599891] Lustre: server umount lustre-MDT0000 complete [ 2385.501577] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 2388.622688] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2399.903965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2399.981149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2399.990297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2400.028249] Lustre: lustre-MDT0000: new disk, initializing [ 2400.046577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2400.053039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2401.287273] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2405.041943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.079569] Lustre: 159352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2405.097379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2405.099162] Lustre: Skipped 1 previous similar message [ 2405.131769] Lustre: lustre-MDT0001: new disk, initializing [ 2405.160409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2405.171788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2405.175483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2406.348236] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2409.916486] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2411.679960] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2413.965485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2414.070358] Lustre: lustre-OST0000: new disk, initializing [ 2414.072891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2414.076367] Lustre: 161319:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2414.111099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.124784] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2417.017341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2417.022222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2417.050980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2419.897820] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2421.750642] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2422.327754] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 16:51:46 (1784235106) [ 2487.263392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2487.266671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.271762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2488.799573] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.799641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2488.806008] Lustre: Skipped 2 previous similar messages [ 2491.102692] Lustre: server umount lustre-MDT0000 complete [ 2492.200396] LustreError: 159344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784235176 with bad export cookie 12347735696328948141 [ 2492.201280] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2492.204738] LustreError: 159344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2493.919455] LustreError: 159358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.919520] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.923719] LustreError: 159358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2493.926813] Lustre: Skipped 1 previous similar message [ 2493.930172] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2497.951484] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2497.951484] LustreError: 162037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.335157] Lustre: server umount lustre-MDT0001 complete [ 2505.610340] Lustre: server umount lustre-OST0000 complete [ 2508.441152] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 2511.269131] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2522.127378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.201034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2522.211959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2522.249870] Lustre: lustre-MDT0000: new disk, initializing [ 2522.269746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.277269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2523.501513] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2527.457569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.494342] Lustre: 165943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2527.512798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2527.514275] Lustre: Skipped 1 previous similar message [ 2527.548896] Lustre: lustre-MDT0001: new disk, initializing [ 2527.587232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2527.590903] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2528.713519] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2532.161360] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2533.876205] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2536.091258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2536.191303] Lustre: lustre-OST0000: new disk, initializing [ 2536.193619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2536.196180] Lustre: 167906:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2536.224162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2536.227201] Lustre: Skipped 1 previous similar message [ 2538.047818] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2539.507193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2539.510970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2539.538862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2541.734950] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2543.186015] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2543.254749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2544.609752] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2544.694753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2549.728685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2549.730157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.732617] LustreError: Skipped 1 previous similar message [ 2549.736624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.739586] Lustre: Skipped 1 previous similar message [ 2551.442147] Lustre: server umount lustre-OST0000 complete [ 2553.311772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2554.850063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.853095] Lustre: Skipped 2 previous similar messages [ 2558.748146] Lustre: server umount lustre-MDT0000 complete [ 2559.968352] LustreError: 167357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.973595] LustreError: 167357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2560.104247] LustreError: 165935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784235244 with bad export cookie 12347735696329106950 [ 2560.106741] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2560.108696] LustreError: 165935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2560.225808] Lustre: server umount lustre-MDT0001 complete [ 2562.984576] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:54:07 (1784235247) [ 2568.375494] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 2571.356886] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2583.548431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.633477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2583.642792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2583.678203] Lustre: lustre-MDT0000: new disk, initializing [ 2583.698820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.705158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2585.009139] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2589.332075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.380579] Lustre: 172654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2589.502130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2590.940551] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2594.787537] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2596.733190] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2599.120363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2599.241592] Lustre: 174633:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2601.273311] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2605.185831] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2606.587685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2606.626230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2616.802213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2616.802597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.806059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2616.809750] Lustre: Skipped 1 previous similar message [ 2616.815517] Lustre: Skipped 4 previous similar messages [ 2618.576870] Lustre: server umount lustre-OST0000 complete [ 2634.207110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2634.269076] Lustre: server umount lustre-MDT0000 complete [ 2635.521472] LustreError: 172649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784235319 with bad export cookie 12347735696329107825 [ 2635.524577] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2635.525668] LustreError: 172649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2635.638127] Lustre: server umount lustre-MDT0001 complete [ 2638.538484] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:55:22 (1784235322) [ 2643.935244] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 2647.068926] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2658.380843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.478034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2658.487455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2658.489211] Lustre: Skipped 3 previous similar messages [ 2658.523486] Lustre: lustre-MDT0000: new disk, initializing [ 2658.524819] Lustre: Skipped 2 previous similar messages [ 2658.544734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2658.546889] Lustre: Skipped 2 previous similar messages [ 2658.551630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2658.554260] Lustre: Skipped 2 previous similar messages [ 2659.901718] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2663.942484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.985505] Lustre: 178950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2664.083759] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2665.366303] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2669.102459] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2670.902354] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2673.116277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2673.217489] Lustre: 180926:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2675.104880] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2678.671187] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2681.876390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2686.946673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2686.946763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2686.950433] Lustre: Skipped 2 previous similar messages [ 2686.950988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2686.952876] LustreError: Skipped 3 previous similar messages [ 2686.957418] Lustre: Skipped 8 previous similar messages [ 2691.732717] Lustre: server umount lustre-OST0000 complete [ 2695.136260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2699.178443] Lustre: server umount lustre-MDT0000 complete [ 2701.265748] LustreError: 178989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784235385 with bad export cookie 12347735696329108896 [ 2701.270444] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2701.272179] LustreError: 178989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2701.478671] Lustre: server umount lustre-MDT0001 complete [ 2706.970493] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2708.352527] Key type lgssc unregistered [ 2708.501727] LNet: 182780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2708.505463] LNetError: 182780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2708.516882] LNet: Removed LNI 192.168.204.128@tcp [ 2708.999900] Key type .llcrypt unregistered [ 2709.001728] Key type ._llcrypt unregistered [ 2723.830559] Key type ._llcrypt registered [ 2723.834593] Key type .llcrypt registered [ 2723.945991] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2724.714209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2724.735068] alg: No test for adler32 (adler32-zlib) [ 2725.729440] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 2725.929113] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2727.615193] Key type lgssc registered [ 2728.795735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2734.406868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2734.419858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2735.678642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.525019] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2741.285051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.468080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2743.623032] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2748.056892] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2752.078468] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2756.846286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.085236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.173783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2761.616147] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2766.187886] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2781.665386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2781.668398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2781.671158] Lustre: Skipped 1 previous similar message [ 2781.676591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2786.787667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2786.791330] Lustre: Skipped 2 previous similar messages [ 2787.058983] Lustre: server umount lustre-OST0000 complete [ 2791.904630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2791.906510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2791.912992] Lustre: Skipped 1 previous similar message [ 2791.920492] Lustre: Skipped 1 previous similar message [ 2797.025314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2797.028345] Lustre: Skipped 1 previous similar message [ 2802.146445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2802.150607] Lustre: Skipped 1 previous similar message [ 2803.167223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2803.258959] Lustre: server umount lustre-MDT0000 complete [ 2804.889172] LustreError: 184157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784235489 with bad export cookie 8806342778363194934 [ 2804.889783] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2804.894778] LustreError: 184157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2805.025929] Lustre: server umount lustre-MDT0001 complete [ 2809.088776] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 2810.380662] Key type lgssc unregistered [ 2810.534754] LNet: 187727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2810.538782] LNetError: 187727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2810.549457] LNet: Removed LNI 192.168.204.128@tcp [ 2810.960823] Key type .llcrypt unregistered [ 2810.962876] Key type ._llcrypt unregistered [ 2823.294184] Key type ._llcrypt registered [ 2823.296263] Key type .llcrypt registered [ 2823.361736] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2824.115562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2824.151944] alg: No test for adler32 (adler32-zlib) [ 2825.214698] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 2825.479147] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 2827.263267] Key type lgssc registered [ 2828.552952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2835.824330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2836.073489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.520230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2843.222364] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2847.712387] LustreError: 189122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.730946] LustreError: 189122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2852.499576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.833125] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2853.029483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2857.516888] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2865.846451] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2872.116717] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2875.755814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2875.874410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.696059] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2879.841782] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2882.022868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2887.137711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.138382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2887.142451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2887.146692] Lustre: Skipped 1 previous similar message [ 2890.257827] Lustre: lustre-OST0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 2893.200078] Lustre: server umount lustre-OST0000 complete [ 2895.861098] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d3dc8f3f800.ost_server_uuid 50 [ 2896.398448] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2912.746332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2912.759714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2912.764456] Lustre: Skipped 2 previous similar messages [ 2912.783912] Lustre: Skipped 3 previous similar messages [ 2916.606586] Lustre: server umount lustre-MDT0000 complete [ 2916.896563] LustreError: 189122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.811598] LustreError: 190965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784235605 with bad export cookie 13541512181321575676 [ 2920.812610] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2920.830254] LustreError: 190965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2921.307922] Lustre: server umount lustre-MDT0001 complete [ 2929.689791] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:00:12 (1784235612) [ 2936.837142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.191997] LustreError: 193034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.236037] LustreError: 193034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2937.317322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2941.553118] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2942.440061] LustreError: 193035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.559804] LustreError: 193034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2949.569445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2949.974425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2953.376971] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2959.635621] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2963.856470] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2969.533321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2969.786454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.415670] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2978.172013] hrtimer: interrupt took 11620081 ns [ 2980.503644] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2982.068728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2986.532244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2986.717850] Lustre: lustre-OST0001: new disk, initializing [ 2986.721551] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2986.726616] Lustre: 195891:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2986.822717] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2992.620802] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2993.246577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2993.253094] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2993.377084] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3001.320891] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3003.879896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3003.890598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3003.899721] Lustre: Skipped 1 previous similar message [ 3003.912682] Lustre: Skipped 1 previous similar message [ 3012.112345] Lustre: lustre-OST0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 3012.121268] Lustre: Skipped 3 previous similar messages [ 3018.207185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3018.276075] Lustre: server umount lustre-OST0000 complete [ 3019.237220] LustreError: 196520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3023.710680] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3023.879464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3027.678952] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3027.915169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3037.720968] LustreError: 194857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.745141] LustreError: 194857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3044.837658] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3044.859441] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.869135] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3044.881075] Lustre: Skipped 1 previous similar message [ 3044.887533] Lustre: Skipped 3 previous similar messages [ 3045.675831] Lustre: server umount lustre-OST0001 complete [ 3052.521650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3052.550873] LustreError: Skipped 1 previous similar message [ 3052.563150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3057.917633] Lustre: server umount lustre-MDT0000 complete [ 3062.248842] LustreError: 193016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784235746 with bad export cookie 13541512181321576740 [ 3062.249181] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3062.259125] LustreError: 193016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3062.633708] Lustre: server umount lustre-MDT0001 complete [ 3084.675293] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:02:48 (1784235768) [ 3090.019319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3090.249862] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3090.290412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3090.433196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3094.130468] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3101.374521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.459279] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3101.499387] Lustre: 199328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3105.577872] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3111.645533] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3116.120539] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3121.684178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3121.809859] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3127.345087] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3129.945491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3134.356982] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3140.293752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3140.336936] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3146.022646] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3152.488765] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3155.425028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3155.425065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.433524] LustreError: Skipped 1 previous similar message [ 3155.436116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.436123] Lustre: Skipped 4 previous similar messages [ 3155.473655] Lustre: Skipped 2 previous similar messages [ 3160.742864] Lustre: server umount lustre-OST0000 complete [ 3162.088096] LustreError: 201681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.111343] LustreError: 201681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3176.929147] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3176.939106] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3180.814044] Lustre: server umount lustre-OST0001 complete [ 3190.151614] Lustre: server umount lustre-MDT0000 complete [ 3193.394641] LustreError: 199318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784235877 with bad export cookie 13541512181321578154 [ 3193.402662] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3193.408181] LustreError: 199318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3193.722645] Lustre: server umount lustre-MDT0001 complete [ 3212.796651] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:04:56 (1784235896) [ 3226.677319] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3234.236704] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3256.166409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.374707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3256.397560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3256.479590] Lustre: lustre-MDT0000: new disk, initializing [ 3256.549799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3256.558201] Lustre: Skipped 3 previous similar messages [ 3256.576143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3260.327563] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3270.017758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3270.176048] Lustre: 207419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3270.263340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3270.273422] Lustre: Skipped 1 previous similar message [ 3270.390098] Lustre: lustre-MDT0001: new disk, initializing [ 3270.590822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3270.622216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3276.509770] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3286.582820] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3292.788574] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3299.022126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3299.260791] Lustre: lustre-OST0000: new disk, initializing [ 3299.268433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3299.274833] Lustre: 209379:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3304.813355] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3307.075878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3307.084408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3307.143097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3312.154289] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3316.150790] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3316.333832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3319.866423] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3320.126178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3322.338965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3322.346668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3322.360128] Lustre: Skipped 4 previous similar messages [ 3322.370564] Lustre: Skipped 10 previous similar messages [ 3336.160490] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3336.298406] Lustre: server umount lustre-OST0000 complete [ 3342.304573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3342.328231] LustreError: Skipped 1 previous similar message [ 3348.268327] Lustre: server umount lustre-MDT0000 complete [ 3353.060254] LustreError: 207430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.099802] LustreError: 207430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3353.360667] LustreError: 207410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784236037 with bad export cookie 13541512181321579715 [ 3353.364947] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3353.384885] LustreError: 207410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3354.140283] Lustre: server umount lustre-MDT0001 complete [ 3367.128263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3374.857774] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3384.437679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.884130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3384.889825] Lustre: Skipped 3 previous similar messages [ 3390.021703] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3398.765768] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3404.133920] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3410.694461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3417.591157] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3425.222247] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3429.549488] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3429.763326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3434.211448] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3434.434387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3438.048762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3438.057040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.074908] Lustre: Skipped 2 previous similar messages [ 3442.535182] Lustre: server umount lustre-OST0000 complete [ 3449.020459] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3453.602078] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3483.048137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3488.636711] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3491.586758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3491.610337] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3496.015316] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3500.815598] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3505.955594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3505.959469] Lustre: Skipped 11 previous similar messages [ 3511.217571] Lustre: server umount lustre-OST0000 complete [ 3521.331216] Lustre: server umount lustre-MDT0000 complete [ 3521.504663] LustreError: 212317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3521.532567] LustreError: 212317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3525.071323] LustreError: 211570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784236209 with bad export cookie 13541512181321580597 [ 3525.084776] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3525.092719] LustreError: 211570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3525.550962] Lustre: server umount lustre-MDT0001 complete [ 3534.666762] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:10:17 (1784236217) [ 3541.414379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.471204] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3555.430911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.935836] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3567.288256] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3573.442036] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3582.195348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3589.089111] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3595.842047] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3600.075825] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3600.320934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3604.873297] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3605.087578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3611.655187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3611.854650] Lustre: lustre-OST0001: new disk, initializing [ 3611.858513] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3611.864286] Lustre: 220894:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3618.602501] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3619.958918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3619.969418] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3620.081650] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3628.358814] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3633.750965] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3633.957305] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3638.458751] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3638.798536] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3645.926198] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3645.948907] LustreError: Skipped 1 previous similar message [ 3645.957306] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.978503] Lustre: Skipped 5 previous similar messages [ 3647.191940] Lustre: server umount lustre-OST0001 complete [ 3653.780433] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3653.975872] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3658.874906] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3659.118250] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3688.919432] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3689.217382] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3689.224134] Lustre: Skipped 6 previous similar messages [ 3694.597765] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3694.614349] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3694.619236] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3694.642694] Lustre: Skipped 1 previous similar message [ 3695.718344] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3703.784129] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3705.211713] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3709.930797] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3709.950776] LustreError: Skipped 1 previous similar message [ 3721.183132] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3721.335639] Lustre: server umount lustre-OST0001 complete [ 3743.588891] Lustre: server umount lustre-OST0000 complete [ 3753.379107] Lustre: server umount lustre-MDT0000 complete [ 3756.917616] LustreError: 220893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784236441 with bad export cookie 13541512181321581556 [ 3756.931832] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3756.933713] LustreError: 220893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3771.359203] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3771.663323] Lustre: server umount lustre-MDT0001 complete [ 3792.822184] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:14:36 (1784236476) [ 3798.782112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.045267] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3799.084253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3803.196797] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3810.796619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.891733] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3810.915722] Lustre: 226116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3815.113482] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3821.449938] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3826.723800] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3833.451406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3833.598661] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3839.935936] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3847.318966] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3856.315413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3856.393405] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3862.934846] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3869.770772] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3871.174512] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3875.537222] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3875.701600] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3879.915378] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3880.154847] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3885.567547] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d3dc5d00000.ost_server_uuid 50 [ 3887.104950] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d3dc5d00000.ost_server_uuid in IDLE state after 0 sec [ 3888.059775] Lustre: Permanently deactivating lustre-OST0001 [ 3888.079573] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3891.888633] Lustre: Permanently reactivating lustre-OST0001 [ 3896.291779] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3896.296269] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3896.310503] LustreError: Skipped 4 previous similar messages [ 3896.336248] Lustre: Skipped 16 previous similar messages [ 3901.737736] Lustre: server umount lustre-OST0001 complete [ 3906.533572] LustreError: 227956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.533578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3906.533589] LustreError: 227956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 35 previous similar messages [ 3906.556777] Lustre: Skipped 8 previous similar messages [ 3911.632338] Lustre: server umount lustre-OST0000 complete [ 3921.470059] Lustre: server umount lustre-MDT0000 complete [ 3925.436523] LustreError: 227971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784236609 with bad export cookie 13541512181321582844 [ 3925.438870] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3925.457677] LustreError: 227971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3925.816827] Lustre: server umount lustre-MDT0001 complete [ 3949.825343] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:17:13 (1784236633) [ 3957.489994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3957.809115] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3957.894280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3957.902990] Lustre: Skipped 5 previous similar messages [ 3962.808548] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3971.864824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3971.972755] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3972.009645] Lustre: 232785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3977.198640] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3985.979632] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3993.392902] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4001.738693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4001.938495] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4001.964735] Lustre: Permanently deactivating lustre-OST0000 [ 4001.986727] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4008.759671] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4011.605933] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4016.971681] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4025.422646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4025.535456] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4035.330197] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4035.420275] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4044.737157] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4046.777287] Lustre: Permanently reactivating lustre-OST0000 [ 4046.795410] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4046.803605] Lustre: Skipped 2 previous similar messages [ 4055.112295] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4055.113503] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 44 seconds. I think it's dead, and I am evicting it. exp 00000000507ffdb3, cur 1784236739 deadline 1784236725 last 1784236695 [ 4055.128115] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4055.163387] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4055.248519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4076.000023] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4076.120094] Lustre: server umount lustre-OST0001 complete [ 4086.091310] Lustre: server umount lustre-OST0000 complete [ 4096.395508] Lustre: server umount lustre-MDT0000 complete [ 4099.869271] LustreError: 232776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784236784 with bad export cookie 13541512181321584496 [ 4099.874649] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4099.887563] LustreError: 232776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4100.366590] Lustre: server umount lustre-MDT0001 complete [ 4112.683338] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4115.888559] Key type lgssc unregistered [ 4116.194520] LNet: 237650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4116.200900] LNetError: 237650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4116.221750] LNet: Removed LNI 192.168.204.128@tcp [ 4117.240583] Key type .llcrypt unregistered [ 4117.242668] Key type ._llcrypt unregistered [ 4139.761845] Key type ._llcrypt registered [ 4139.764491] Key type .llcrypt registered [ 4139.860720] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4140.855428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4140.935959] alg: No test for adler32 (adler32-zlib) [ 4142.072405] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 4142.297868] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 4143.951571] Key type lgssc registered [ 4145.034101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4155.055458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4155.096883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4156.525965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.763872] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4166.127475] LustreError: 239112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4166.153590] LustreError: 239112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4169.658684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4169.836850] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4169.877264] Lustre: 239100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4169.891702] Lustre: Permanently deactivating lustre-MDT0001 [ 4169.918295] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4170.384909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4171.892436] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4171.913819] LustreError: 239143:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4171.929200] LustreError: 239143:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4171.943565] LustreError: 239143:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 4175.046378] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4183.700720] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4190.555528] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4200.228537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4200.945380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4201.956503] LustreError: 240951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.013388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4207.087080] LustreError: 240951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4208.835696] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4212.201382] LustreError: 241000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4217.626978] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4220.395562] LustreError: 240948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.430065] LustreError: 240948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4226.494113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4226.673774] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4232.189597] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4232.209462] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4233.984601] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4242.791730] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4246.122675] Lustre: Permanently reactivating lustre-MDT0001 [ 4246.147489] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4246.150816] Lustre: Skipped 2 previous similar messages [ 4250.686946] Lustre: lustre-MDT0001: Client c0fed74f-f081-484b-a3f9-8003d8857cc6 (at 192.168.204.28@tcp) reconnecting [ 4257.798820] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4260.106199] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4267.238965] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4267.539567] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4272.219974] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4272.435641] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4274.892533] Lustre: Permanently deactivating lustre-MDT0001 [ 4274.904256] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4274.910134] Lustre: Skipped 2 previous similar messages [ 4282.024959] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4291.556645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.559636] LustreError: 239108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4291.582315] Lustre: Skipped 3 previous similar messages [ 4291.624106] LustreError: 239108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4291.730902] Lustre: server umount lustre-MDT0000 complete [ 4296.081737] LustreError: 239093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784236980 with bad export cookie 6050305496420198331 [ 4296.099057] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4296.116871] LustreError: 239093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4296.572348] Lustre: server umount lustre-MDT0001 complete [ 4302.785318] Lustre: server umount lustre-OST0000 complete [ 4315.441727] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:23:18 (1784236998) [ 4325.365434] Lustre: server umount lustre-OST0001 complete [ 4332.312052] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 4341.067344] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4371.521269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4371.769601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4371.776370] Lustre: Skipped 2 previous similar messages [ 4371.791153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4371.870286] Lustre: lustre-MDT0000: new disk, initializing [ 4371.947683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4371.960567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4375.409706] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4385.399776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4385.500533] Lustre: 247342:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4385.569102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4385.573847] Lustre: Skipped 1 previous similar message [ 4385.687787] Lustre: lustre-MDT0001: new disk, initializing [ 4385.794543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4385.845770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4385.854270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4390.149697] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4398.976364] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4405.341982] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4412.144340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4412.422574] Lustre: lustre-OST0000: new disk, initializing [ 4412.425581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4412.431962] Lustre: 249303:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4412.509151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4419.103130] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4420.221147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4420.253265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4420.406102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4430.582952] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4432.943083] LustreError: 250019:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4442.618947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4442.886030] Lustre: lustre-OST0001: new disk, initializing [ 4442.895594] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4442.903709] Lustre: 250415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4442.980370] LustreError: 250019:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4443.038188] LustreError: 247348:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4449.948950] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4450.966298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4450.982484] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4451.118461] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4453.119234] LustreError: 247348:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4461.369373] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4463.388027] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4466.659032] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4466.678453] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4466.684452] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4466.692455] Lustre: Skipped 1 previous similar message [ 4466.717426] Lustre: Skipped 1 previous similar message [ 4471.788907] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4471.796651] Lustre: Skipped 1 previous similar message [ 4472.586220] Lustre: server umount lustre-OST0001 complete [ 4476.900079] LustreError: 249287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4476.923866] LustreError: 249287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4492.266841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4492.267125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4492.274757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4492.277125] LustreError: Skipped 1 previous similar message [ 4492.302537] Lustre: Skipped 1 previous similar message [ 4495.405541] Lustre: server umount lustre-OST0000 complete [ 4502.499516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4502.507617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4502.513334] Lustre: Skipped 1 previous similar message [ 4502.518793] Lustre: Skipped 1 previous similar message [ 4505.520568] Lustre: server umount lustre-MDT0000 complete [ 4508.746537] LustreError: 249302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784237193 with bad export cookie 6050305496420200928 [ 4508.752192] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4508.762303] LustreError: 249302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4515.116181] Lustre: server umount lustre-MDT0001 complete [ 4524.764908] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4527.069757] Key type lgssc unregistered [ 4527.400941] LNet: 252439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4527.415495] LNetError: 252439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4527.441440] LNet: Removed LNI 192.168.204.128@tcp [ 4528.368753] Key type .llcrypt unregistered [ 4528.371825] Key type ._llcrypt unregistered [ 4569.964227] Key type ._llcrypt registered [ 4569.966872] Key type .llcrypt registered [ 4570.097904] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4571.053706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4571.084852] alg: No test for adler32 (adler32-zlib) [ 4572.196577] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 4572.523538] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 4574.223467] Key type lgssc registered [ 4575.678577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4584.531407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4584.587887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4585.875095] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4585.914979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4586.144204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4591.102066] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4598.402376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4598.501468] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4598.552885] Lustre: 254640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4598.818041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4603.016412] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4610.572804] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4617.150476] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4624.368372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4624.592294] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4624.768852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4631.190727] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4634.740155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4637.953702] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4690.913027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4690.925127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.939247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4691.425154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4691.444497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4691.469205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4691.477765] Lustre: Skipped 1 previous similar message [ 4695.279020] Lustre: server umount lustre-OST0000 complete [ 4700.338882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4713.664466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4714.109164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4716.143302] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4716.163439] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4720.478845] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4728.034834] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4731.565531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4731.583471] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4767.713717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.718858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4767.745473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4767.757354] Lustre: Skipped 1 previous similar message [ 4772.642888] Lustre: server umount lustre-OST0000 complete [ 4777.957351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4777.958085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.991290] Lustre: Skipped 1 previous similar message [ 4778.012543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4778.020869] Lustre: Skipped 1 previous similar message [ 4782.393180] Lustre: server umount lustre-MDT0000 complete [ 4785.696348] LustreError: 254631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784237469 with bad export cookie 1675659896046623231 [ 4785.701899] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4785.710442] LustreError: 254631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4786.011311] Lustre: server umount lustre-MDT0001 complete [ 4795.521936] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4798.078983] Key type lgssc unregistered [ 4798.431273] LNet: 260036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4798.447019] LNetError: 260036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4798.473060] LNet: Removed LNI 192.168.204.128@tcp [ 4799.264424] Key type .llcrypt unregistered [ 4799.266613] Key type ._llcrypt unregistered [ 4820.762593] Key type ._llcrypt registered [ 4820.765247] Key type .llcrypt registered [ 4820.861486] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4821.656961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4821.760225] alg: No test for adler32 (adler32-zlib) [ 4822.795690] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 4822.992057] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 4824.647166] Key type lgssc registered [ 4825.701702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4832.648701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4832.666161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.095389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4838.684608] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4844.526115] LustreError: 261455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4844.554067] LustreError: 261455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4846.726885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4847.004889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4850.655470] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4857.763418] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4863.020272] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4869.543564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4869.803624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4876.080715] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4879.033122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4884.242874] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4908.016205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4908.025058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4908.041402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4910.056311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4910.061717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4910.062766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4910.095411] Lustre: Skipped 1 previous similar message [ 4912.154958] Lustre: server umount lustre-OST0000 complete [ 4918.753066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4918.761484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4918.779705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4922.447734] Lustre: server umount lustre-MDT0000 complete [ 4925.419074] LustreError: 261455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4925.435714] LustreError: 261455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4926.026800] LustreError: 261434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784237610 with bad export cookie 14626734898802523644 [ 4926.033356] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4926.038730] LustreError: 261434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4926.442135] Lustre: server umount lustre-MDT0001 complete [ 4936.137682] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4938.768068] Key type lgssc unregistered [ 4939.158991] LNet: 265744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4939.169643] LNetError: 265744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4939.182855] LNet: Removed LNI 192.168.204.128@tcp [ 4940.045782] Key type .llcrypt unregistered [ 4940.048190] Key type ._llcrypt unregistered [ 4956.482393] Key type ._llcrypt registered [ 4956.487667] Key type .llcrypt registered [ 4956.640794] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4957.759214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4957.830114] alg: No test for adler32 (adler32-zlib) [ 4958.903451] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 4959.215177] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 4960.935457] Key type lgssc registered [ 4962.074918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4969.472195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4969.500253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.991451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4975.291272] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4981.220061] LustreError: 266961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4981.232041] LustreError: 266961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4983.060389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4983.508182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4987.818215] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4994.371412] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4999.443514] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5006.216403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5006.535387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5012.712615] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5015.804330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5019.955346] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5031.408493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5031.418359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5031.425910] Lustre: Skipped 1 previous similar message [ 5031.428061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5031.431083] Lustre: Skipped 1 previous similar message [ 5036.514719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5036.521677] Lustre: Skipped 1 previous similar message [ 5037.701248] Lustre: server umount lustre-OST0000 complete [ 5041.633685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5041.646701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5041.647716] Lustre: Skipped 1 previous similar message [ 5041.665678] Lustre: Skipped 1 previous similar message [ 5046.755895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5046.758630] Lustre: Skipped 1 previous similar message [ 5047.362890] Lustre: server umount lustre-MDT0000 complete [ 5050.957224] LustreError: 266941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784237735 with bad export cookie 17060405162331196356 [ 5050.962802] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5050.971280] LustreError: 266941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5051.374801] Lustre: server umount lustre-MDT0001 complete [ 5062.036550] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5064.902690] Key type lgssc unregistered [ 5065.211601] LNet: 270626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5065.223042] LNetError: 270626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5066.283427] LNet: Removed LNI 192.168.204.128@tcp [ 5067.117917] Key type .llcrypt unregistered [ 5067.120096] Key type ._llcrypt unregistered [ 5094.009258] Key type ._llcrypt registered [ 5094.012913] Key type .llcrypt registered [ 5094.091088] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5095.404894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5095.461761] alg: No test for adler32 (adler32-zlib) [ 5096.575983] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 5096.849988] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5098.535914] Key type lgssc registered [ 5100.038188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5108.454781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5108.467354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5109.973632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5114.785945] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5120.494248] LustreError: 272039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5120.507915] LustreError: 272039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5123.646577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5124.031960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5128.716886] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5136.118473] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5142.070611] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5148.952280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5149.330727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5155.519351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5155.995233] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5163.923967] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5191.656393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.658746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5191.678976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5194.673040] Lustre: server umount lustre-OST0000 complete [ 5200.864470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5200.875208] LustreError: Skipped 1 previous similar message [ 5200.881728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.892860] Lustre: Skipped 1 previous similar message [ 5200.906841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5200.916862] Lustre: Skipped 1 previous similar message [ 5201.889589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5201.946615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5201.953958] Lustre: Skipped 1 previous similar message [ 5205.330312] Lustre: server umount lustre-MDT0000 complete [ 5207.008621] LustreError: 272034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5207.034770] LustreError: 272034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5209.718639] LustreError: 272019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784237893 with bad export cookie 6619877099143187378 [ 5209.725512] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5209.727638] LustreError: 272019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5210.175582] Lustre: server umount lustre-MDT0001 complete [ 5222.962555] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5225.590560] Key type lgssc unregistered [ 5225.919446] LNet: 276331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5225.937874] LNetError: 276331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5225.969067] LNet: Removed LNI 192.168.204.128@tcp [ 5226.861092] Key type .llcrypt unregistered [ 5226.864408] Key type ._llcrypt unregistered [ 5246.539638] Key type ._llcrypt registered [ 5246.545599] Key type .llcrypt registered [ 5246.760163] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5248.064461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5248.292123] alg: No test for adler32 (adler32-zlib) [ 5249.361481] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 5249.685911] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5251.439205] Key type lgssc registered [ 5252.890683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5261.808730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5261.860838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5263.594653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5268.289719] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5273.058926] LustreError: 277563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5273.080435] LustreError: 277563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5276.593507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5277.054872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5281.618494] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5290.728753] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5296.807718] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5304.966452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5305.348599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5308.411693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5311.391597] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5319.008573] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5339.111094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5339.112421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5339.131551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5339.148901] LustreError: Skipped 1 previous similar message [ 5339.171794] Lustre: Skipped 1 previous similar message [ 5341.153947] Lustre: server umount lustre-OST0000 complete [ 5348.833370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5348.842171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5348.862974] Lustre: Skipped 1 previous similar message [ 5348.872861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5351.548586] Lustre: server umount lustre-MDT0000 complete [ 5354.472515] LustreError: 277562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5354.492160] LustreError: 277562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5355.594184] LustreError: 277528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784238039 with bad export cookie 13757380791332962646 [ 5355.595775] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5355.605091] LustreError: 277528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5355.934299] Lustre: server umount lustre-MDT0001 complete [ 5368.485501] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5371.693961] Key type lgssc unregistered [ 5372.075289] LNet: 281259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5372.082394] LNetError: 281259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5372.103549] LNet: Removed LNI 192.168.204.128@tcp [ 5373.343943] Key type .llcrypt unregistered [ 5373.346814] Key type ._llcrypt unregistered [ 5386.783414] Key type ._llcrypt registered [ 5386.785725] Key type .llcrypt registered [ 5386.922815] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5402.997224] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 5419.222809] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5421.027766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5421.129134] alg: No test for adler32 (adler32-zlib) [ 5422.290388] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 5422.638638] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5424.407256] Key type lgssc registered [ 5425.982176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5467.821942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5467.847729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5469.163787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5469.222220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5469.309709] Lustre: lustre-MDT0000: new disk, initializing [ 5469.379715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5469.395312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5473.322395] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5480.798597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5480.881555] Lustre: 284810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5480.928571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5480.932979] Lustre: Skipped 1 previous similar message [ 5480.987914] Lustre: lustre-MDT0001: new disk, initializing [ 5481.066085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5481.091859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5481.100479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5484.102618] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5493.798914] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5499.589440] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5505.697745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5505.974560] Lustre: lustre-OST0000: new disk, initializing [ 5505.978589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5505.987272] Lustre: 286774:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5506.081952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.834708] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5515.331741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5515.350030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5515.487664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5521.644552] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5526.361044] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5526.654034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5532.326216] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5532.628509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5536.237851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5536.238230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5536.253905] LustreError: Skipped 1 previous similar message [ 5536.269730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5536.289939] Lustre: Skipped 1 previous similar message [ 5536.317440] Lustre: Skipped 1 previous similar message [ 5541.006088] Lustre: server umount lustre-OST0000 complete [ 5546.464958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5546.477147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5546.484420] Lustre: Skipped 1 previous similar message [ 5546.506281] Lustre: Skipped 1 previous similar message [ 5551.519938] Lustre: server umount lustre-MDT0000 complete [ 5556.240966] LustreError: 285602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784238240 with bad export cookie 11267086475940979168 [ 5556.251769] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5556.262288] LustreError: 285602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5556.671436] Lustre: server umount lustre-MDT0001 complete [ 5566.209125] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:44:09 (1784238249) [ 5573.501726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5574.017265] LustreError: 289176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.035852] LustreError: 289176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5574.091779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5579.209301] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5579.232710] LustreError: 289177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5584.364371] LustreError: 289176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5590.108552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5590.496919] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5590.781499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5596.801585] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5605.557919] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5612.851126] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5620.192605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5620.558975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5627.842102] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5636.475323] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5650.405720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5650.411441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5650.430323] Lustre: Skipped 1 previous similar message [ 5650.447247] Lustre: Skipped 1 previous similar message [ 5654.130567] Lustre: server umount lustre-OST0000 complete [ 5660.650428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5660.660113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5660.693031] Lustre: Skipped 1 previous similar message [ 5660.720838] Lustre: Skipped 1 previous similar message [ 5665.421511] Lustre: server umount lustre-MDT0000 complete [ 5665.761268] LustreError: 290114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5665.798921] LustreError: 290114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5669.265537] LustreError: 289157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784238353 with bad export cookie 11267086475940980050 [ 5669.269751] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5669.317911] LustreError: 289157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5669.796816] Lustre: server umount lustre-MDT0001 complete [ 5682.488539] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5685.772029] Key type lgssc unregistered [ 5686.223736] LNet: 292697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5686.236651] LNetError: 292697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5686.258570] LNet: Removed LNI 192.168.204.128@tcp [ 5687.126337] Key type .llcrypt unregistered [ 5687.130864] Key type ._llcrypt unregistered [ 5715.077762] Key type ._llcrypt registered [ 5715.080860] Key type .llcrypt registered [ 5715.233411] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 5732.462583] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5733.718783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5733.769815] alg: No test for adler32 (adler32-zlib) [ 5735.099119] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 5735.485570] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5737.215989] Key type lgssc registered [ 5738.416965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5786.437858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5786.491477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.909651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5787.964036] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5788.152132] Lustre: lustre-MDT0000: new disk, initializing [ 5788.290498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5788.311386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5794.569958] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5807.635055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5807.778759] Lustre: 296511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5807.843603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5807.850059] Lustre: Skipped 1 previous similar message [ 5807.992721] Lustre: lustre-MDT0001: new disk, initializing [ 5808.166325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5808.231461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5808.254199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5814.050865] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5824.440620] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5830.237636] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5838.584801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5838.965991] Lustre: lustre-OST2710: new disk, initializing [ 5838.974555] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5838.987234] Lustre: 298474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5839.202622] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5846.172561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5846.191110] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5846.445665] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5846.639271] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5858.386260] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5866.236586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5866.465517] Lustre: lustre-OST03e8: new disk, initializing [ 5866.474796] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5866.494466] Lustre: 299535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5866.602558] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5873.278725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5873.292565] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5873.405983] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5874.468959] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5883.193597] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5891.225593] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5891.478935] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5896.317370] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5896.564621] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5903.119888] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5903.353797] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5909.737709] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5910.157336] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5919.715756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5919.750625] Lustre: Skipped 1 previous similar message [ 5919.750879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5919.780029] Lustre: Skipped 2 previous similar messages [ 5924.203418] Lustre: server umount lustre-MDT0000 complete [ 5929.956582] LustreError: 297671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5930.015978] LustreError: 297671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5930.433511] LustreError: 296504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784238614 with bad export cookie 7177458509759053080 [ 5930.453154] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5932.147591] Lustre: server umount lustre-MDT0001 complete [ 5950.356089] Lustre: server umount lustre-OST2710 complete [ 5951.521688] Lustre: 294316:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784238619/real 1784238619] req@00000000a7441883 x1870909584517504/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784238635 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5951.589549] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5951.616277] Lustre: Skipped 1 previous similar message [ 5955.745207] Lustre: 294316:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784238624/real 1784238624] req@00000000ec5a9175 x1870909584517760/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784238640 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5956.583139] Lustre: server umount lustre-OST03e8 complete [ 5981.980296] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 5994.179664] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6038.639781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6039.022679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6039.080993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6039.279347] Lustre: lustre-MDT0000: new disk, initializing [ 6039.412652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6039.434553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6044.564075] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6056.440691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6056.665711] Lustre: 304764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6056.757492] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6056.763414] Lustre: Skipped 1 previous similar message [ 6056.863680] Lustre: lustre-MDT0001: new disk, initializing [ 6057.008925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6057.066096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6057.088302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6062.497884] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6075.099979] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6082.721160] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6090.829822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6091.251508] Lustre: lustre-OST0000: new disk, initializing [ 6091.287426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6091.301455] Lustre: 306731:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6091.405112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6096.972315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6096.983683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6097.127882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6098.679533] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6108.502670] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6114.060184] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6114.295951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6119.583851] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6119.935899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6122.983917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6122.994306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.998230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6123.012472] Lustre: Skipped 1 previous similar message [ 6123.028696] Lustre: Skipped 2 previous similar messages [ 6128.108638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6128.121385] Lustre: Skipped 1 previous similar message [ 6133.218772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6133.234318] Lustre: Skipped 1 previous similar message [ 6136.287175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6136.514935] Lustre: server umount lustre-OST0000 complete [ 6143.458581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6143.463994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6143.471125] Lustre: Skipped 1 previous similar message [ 6143.484738] Lustre: Skipped 1 previous similar message [ 6146.435543] Lustre: server umount lustre-MDT0000 complete [ 6148.580161] LustreError: 304776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.595670] LustreError: 304776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6149.876419] LustreError: 304756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784238834 with bad export cookie 7177458509759054781 [ 6149.880507] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6149.889374] LustreError: 304756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6150.248850] Lustre: server umount lustre-MDT0001 complete [ 6158.661277] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:54:01 (1784238841) [ 6160.140553] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6162.189344] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:54:05 (1784238845) [ 6165.141635] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6199.523712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6199.822747] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6199.837555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6200.063501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6203.981737] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6214.260605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.429354] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6214.503498] Lustre: 310575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6214.806966] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6220.339055] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6242.179269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6242.383073] Lustre: Denying initial registration attempt from nid 192.168.204.128@tcp, specified as failover [ 6242.389703] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6242.396691] LustreError: 312024:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 6242.401905] LustreError: 312024:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6242.407958] LustreError: 312024:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6242.477436] Lustre: server umount lustre-OST0000 complete [ 6242.485693] LustreError: 312024:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6250.976544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6250.983406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6250.988823] LustreError: Skipped 1 previous similar message [ 6250.989851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6251.027070] Lustre: Skipped 1 previous similar message [ 6254.553571] Lustre: server umount lustre-MDT0000 complete [ 6256.098466] LustreError: 310580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6256.129042] LustreError: 310580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6257.941527] LustreError: 310567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784238942 with bad export cookie 7177458509759055656 [ 6257.945185] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6257.958785] LustreError: 310567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6258.339032] Lustre: server umount lustre-MDT0001 complete [ 6268.430380] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 6270.802669] Key type lgssc unregistered [ 6271.011879] LNet: 313141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6271.019767] LNetError: 313141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6271.032151] LNet: Removed LNI 192.168.204.128@tcp [ 6271.794344] Key type .llcrypt unregistered [ 6271.797372] Key type ._llcrypt unregistered [ 6282.489297] Key type ._llcrypt registered [ 6282.492335] Key type .llcrypt registered [ 6282.622416] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6283.480157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6283.517334] alg: No test for adler32 (adler32-zlib) [ 6284.694494] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 6285.201464] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 6286.993598] Key type lgssc registered [ 6288.368825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6344.878967] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6354.103239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6354.123115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6355.417443] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6355.452083] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6355.698548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6359.888769] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6368.168744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6368.276300] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6368.350853] Lustre: 315894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6368.688223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6373.152443] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6380.118319] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6386.402500] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6406.003520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6406.237812] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6406.465513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6412.526230] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6420.263840] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6422.499060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6422.507544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6422.515873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6422.542064] Lustre: Skipped 1 previous similar message [ 6427.622235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6427.630078] Lustre: Skipped 2 previous similar messages [ 6428.669816] Lustre: server umount lustre-OST0000 complete [ 6432.754613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6432.769951] Lustre: Skipped 1 previous similar message [ 6432.771586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6432.787116] Lustre: Skipped 1 previous similar message [ 6437.856978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6437.866629] Lustre: Skipped 1 previous similar message [ 6439.093144] Lustre: server umount lustre-MDT0000 complete [ 6442.611413] LustreError: 317754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784239126 with bad export cookie 7147771230385626960 [ 6442.612620] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6442.625833] LustreError: 317754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6443.046590] Lustre: server umount lustre-MDT0001 complete [ 6456.235550] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 6459.348308] Key type lgssc unregistered [ 6459.593509] LNet: 319455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6459.602591] LNetError: 319455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6459.622821] LNet: Removed LNI 192.168.204.128@tcp [ 6460.470657] Key type .llcrypt unregistered [ 6460.477536] Key type ._llcrypt unregistered [ 6488.541737] Key type ._llcrypt registered [ 6488.546371] Key type .llcrypt registered [ 6488.750510] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6490.206158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6490.325213] alg: No test for adler32 (adler32-zlib) [ 6491.516239] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 6491.923454] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 6493.663190] Key type lgssc registered [ 6494.913273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6503.834794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6503.869391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6505.683640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6510.700535] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6515.175091] LustreError: 320868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6515.190397] LustreError: 320868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6520.310249] LustreError: 320867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6521.902787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6522.233363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6526.667385] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6535.090976] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6541.781122] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6551.888333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6552.285512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6559.264187] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6567.167822] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6588.898888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6588.906128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.929749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6589.495784] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 6590.445988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6590.469991] Lustre: Skipped 1 previous similar message [ 6592.896971] Lustre: server umount lustre-MDT0000 complete [ 6594.585670] LustreError: 323373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6597.359913] LustreError: 320847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784239281 with bad export cookie 3582731377445108496 [ 6597.365350] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6597.376158] LustreError: 320847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6599.693807] LustreError: 323373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6599.726088] LustreError: 323373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6599.743656] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [ 6599.754503] Lustre: Skipped 3 previous similar messages [ 6600.681539] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6602.733708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6602.741066] Lustre: Skipped 1 previous similar message [ 6603.811813] Lustre: server umount lustre-MDT0001 complete [ 6611.732648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6623.135804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6643.146561] LustreError: 322697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6643.168358] LustreError: 322697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6643.228704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6643.318066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 6648.474861] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6652.400573] LustreError: 324481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6652.444745] LustreError: 324481:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6657.522390] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6658.301498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6658.634492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6663.685504] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6664.390780] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6674.135402] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6676.212629] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6682.912480] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6685.119731] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6694.373470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6694.374675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6694.398486] LustreError: Skipped 1 previous similar message [ 6694.400426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6694.438283] Lustre: Skipped 1 previous similar message [ 6697.720541] Lustre: server umount lustre-OST0000 complete [ 6704.619963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6704.622387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6704.639789] Lustre: Skipped 1 previous similar message [ 6704.645501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6704.670770] Lustre: Skipped 2 previous similar messages [ 6708.778157] Lustre: server umount lustre-MDT0000 complete [ 6709.728626] LustreError: 324485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.750931] LustreError: 324485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6713.052940] LustreError: 320849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784239397 with bad export cookie 3582731377445116084 [ 6713.062389] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6713.070312] LustreError: 320849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6713.547092] Lustre: server umount lustre-MDT0001 complete [ 6726.662649] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 6730.188799] Key type lgssc unregistered [ 6730.674830] LNet: 327144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6730.687301] LNetError: 327144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6730.709437] LNet: Removed LNI 192.168.204.128@tcp [ 6731.753343] Key type .llcrypt unregistered [ 6731.756619] Key type ._llcrypt unregistered [ 6759.827657] Key type ._llcrypt registered [ 6759.830843] Key type .llcrypt registered [ 6759.962218] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 6761.170829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6761.394256] alg: No test for adler32 (adler32-zlib) [ 6762.527576] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 6762.831703] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 6764.543194] Key type lgssc registered [ 6765.876760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6774.839314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6774.856536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6776.543139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6781.781471] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6786.024791] LustreError: 328555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6786.040879] LustreError: 328555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6791.138564] LustreError: 328556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6791.431906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6791.757947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6797.200628] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6805.736621] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6811.643802] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6815.781192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6815.787393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6815.795898] Lustre: Skipped 1 previous similar message [ 6815.812393] Lustre: Skipped 1 previous similar message [ 6820.834429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6820.843867] Lustre: Skipped 1 previous similar message [ 6825.962093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6825.974654] Lustre: Skipped 1 previous similar message [ 6829.023248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6829.291388] Lustre: server umount lustre-MDT0000 complete [ 6831.079188] LustreError: 328555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6831.111321] LustreError: 328555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6833.435170] LustreError: 328536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784239517 with bad export cookie 5119358665084400551 [ 6833.451710] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6833.459958] LustreError: 328536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6833.823347] Lustre: server umount lustre-MDT0001 complete [ 6853.505171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6854.013358] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6854.092274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6854.416272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6859.927213] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6869.932175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6870.046445] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6870.084855] Lustre: 331134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6870.420972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6876.105694] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6883.531131] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6889.044578] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6898.286182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6898.499924] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6898.508121] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6898.522556] LustreError: 333014:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6898.529911] LustreError: 333014:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6898.536223] LustreError: 333014:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6898.588798] Lustre: server umount lustre-OST0000 complete [ 6898.591828] LustreError: 333014:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6907.503778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6907.750659] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6907.955669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6913.253996] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6913.663061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 6921.776281] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6929.625281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6929.752720] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6929.879431] Lustre: lustre-OST0001: new disk, initializing [ 6929.882869] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6929.888917] Lustre: 334550:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6929.976917] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6936.664811] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 6940.271826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6940.285696] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6940.435956] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6946.188451] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6950.886030] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6950.895506] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.897325] LustreError: Skipped 1 previous similar message [ 6950.905274] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6950.916199] Lustre: Skipped 1 previous similar message [ 6950.949640] Lustre: Skipped 1 previous similar message [ 6956.006062] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6956.018213] Lustre: Skipped 1 previous similar message [ 6964.191184] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6964.466607] Lustre: server umount lustre-OST0001 complete [ 6966.242379] LustreError: 333480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6966.278302] LustreError: 333480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6969.826080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6969.845501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6969.866333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6969.882620] Lustre: Skipped 2 previous similar messages [ 6971.364862] LustreError: 333480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6971.384179] LustreError: 333480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6975.724722] Lustre: server umount lustre-OST0000 complete [ 6981.600983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.635352] Lustre: Skipped 2 previous similar messages [ 6986.086449] Lustre: server umount lustre-MDT0000 complete [ 6986.731047] LustreError: 331140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6986.754503] LustreError: 331140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6990.365471] LustreError: 331127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784239674 with bad export cookie 5119358665084401104 [ 6990.373798] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6990.383535] LustreError: 331127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6990.879153] Lustre: server umount lustre-MDT0001 complete [ 7004.943317] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7008.639476] Key type lgssc unregistered [ 7008.899536] LNet: 336500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7008.919492] LNetError: 336500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7008.970459] LNet: Removed LNI 192.168.204.128@tcp [ 7010.249802] Key type .llcrypt unregistered [ 7010.252862] Key type ._llcrypt unregistered [ 7093.475491] Key type ._llcrypt registered [ 7093.477755] Key type .llcrypt registered [ 7093.574100] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 7108.936576] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7110.419814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7110.466115] alg: No test for adler32 (adler32-zlib) [ 7111.648750] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 7111.955140] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 7113.647167] Key type lgssc registered [ 7114.829791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7150.341295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7150.382245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7151.723750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7151.758959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7151.910631] Lustre: lustre-MDT0000: new disk, initializing [ 7152.115191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7152.133349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7157.442448] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7171.712882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7171.996149] Lustre: 341415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7172.102743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7172.110235] Lustre: Skipped 1 previous similar message [ 7172.198536] Lustre: lustre-MDT0001: new disk, initializing [ 7172.364329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7172.436227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7172.463171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7179.148655] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7191.051329] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7196.839254] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7204.243813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7204.646646] Lustre: lustre-OST0000: new disk, initializing [ 7204.654617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7204.666990] Lustre: 343377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7204.806800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7211.610380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7211.630272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7211.818078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7211.878581] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7222.864837] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7229.099447] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7229.460990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7235.693483] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7236.060339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7242.726759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7242.732790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7242.743969] LustreError: Skipped 1 previous similar message [ 7242.746082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7242.822142] Lustre: Skipped 1 previous similar message [ 7245.245791] Lustre: server umount lustre-OST0000 complete [ 7252.969391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.992717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7252.999099] Lustre: Skipped 2 previous similar messages [ 7255.944119] Lustre: server umount lustre-MDT0000 complete [ 7260.335519] LustreError: 341408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784239944 with bad export cookie 16464796287120512604 [ 7260.342328] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7260.348096] LustreError: 341408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7260.772474] Lustre: server umount lustre-MDT0001 complete [ 7269.446937] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 18:12:32 (1784239952) [ 7277.271184] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 18:12:40 (1784239960) [ 7286.013050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7286.618168] LustreError: 346122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7286.640263] LustreError: 346122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7286.708356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7291.147346] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7291.875100] LustreError: 346123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7295.975620] LustreError: 346122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7299.059862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7299.424976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7304.315072] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7311.548665] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7317.119977] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7323.844237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7324.245786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7330.519615] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7337.846472] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7346.361098] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7348.219462] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7350.008188] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7351.965513] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7353.960178] Lustre: Failing over lustre-MDT0000 [ 7354.290167] Lustre: server umount lustre-MDT0000 complete [ 7355.880107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7355.888673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7355.901595] Lustre: Skipped 1 previous similar message [ 7355.911749] LustreError: 347766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7355.960371] LustreError: 347766:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7361.038677] LustreError: 346117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7361.064039] LustreError: 346117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7371.282115] LustreError: 346118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7371.305921] LustreError: 346118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7373.280607] Lustre: 339644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784240041/real 1784240041] req@0000000077ad0de0 x1870911028404096/t0(0) o400->MGC192.168.204.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784240057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7373.320961] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7373.569581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.533240] LustreError: 339643:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ec9af226 x1870911028410112/t0(0) o250->MGC192.168.204.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7383.997543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7384.053679] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7387.669228] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7389.167097] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7389.226617] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 7389.251506] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7389.284150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7399.482774] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7401.010786] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7404.828913] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7406.980453] Lustre: Failing over lustre-MDT0000 [ 7407.454502] Lustre: server umount lustre-MDT0000 complete [ 7408.197745] LustreError: 346117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7408.223796] LustreError: 346117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 7409.633990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7409.635955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7409.661357] Lustre: Skipped 3 previous similar messages [ 7426.015881] Lustre: 339645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784240093/real 1784240093] req@0000000064729082 x1870911028425472/t0(0) o400->MGC192.168.204.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784240109 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7426.089544] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7428.692950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7436.273041] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe47eb4fd28fd20f8 [ 7436.285717] Lustre: MGC192.168.204.128@tcp: Connection restored to 0@lo (at 0@lo) [ 7436.297611] Lustre: Skipped 2 previous similar messages [ 7436.494882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7436.547961] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7439.898935] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7441.342757] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7441.911589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7441.957384] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 7442.012673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7451.606757] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7453.409960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7462.369632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7462.371420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7462.372811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7462.406071] Lustre: Skipped 3 previous similar messages [ 7462.433068] Lustre: Skipped 2 previous similar messages [ 7466.274418] Lustre: server umount lustre-MDT0000 complete [ 7467.494170] LustreError: 346122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7467.515095] LustreError: 346122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 28 previous similar messages [ 7470.329478] LustreError: 348885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784240154 with bad export cookie 16464796287120515320 [ 7470.335765] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7470.342639] LustreError: 348885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7470.754580] Lustre: server umount lustre-MDT0001 complete [ 7485.529067] Lustre: server umount lustre-OST0000 complete [ 7501.588869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7502.182558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7506.788257] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7516.329539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7521.680972] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7530.686502] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7538.262933] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7547.943642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7556.423072] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7564.180352] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7572.218811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7581.665251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7581.679366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7581.709931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7586.481968] Lustre: server umount lustre-OST0000 complete [ 7592.931714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7592.950582] Lustre: Skipped 3 previous similar messages [ 7596.945844] Lustre: server umount lustre-MDT0000 complete [ 7598.056613] LustreError: 352437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7598.087090] LustreError: 352437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7601.085506] LustreError: 352419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784240285 with bad export cookie 16464796287120516069 [ 7601.105779] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7601.488467] Lustre: server umount lustre-MDT0001 complete [ 7612.980810] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7616.082345] Key type lgssc unregistered [ 7616.448635] LNet: 355975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7616.459354] LNetError: 355975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7616.474533] LNet: Removed LNI 192.168.204.128@tcp [ 7617.391179] Key type .llcrypt unregistered [ 7617.397545] Key type ._llcrypt unregistered [ 7643.681914] Key type ._llcrypt registered [ 7643.688500] Key type .llcrypt registered [ 7643.851431] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7645.173103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7645.199851] alg: No test for adler32 (adler32-zlib) [ 7646.333906] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 7646.690022] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 7648.407215] Key type lgssc registered [ 7649.576784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7658.057194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7658.075947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.667974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7664.473028] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7669.229041] LustreError: 357489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.250632] LustreError: 357489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7674.083274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7674.341786] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7674.477596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7679.190493] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7687.228700] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7694.193784] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7702.466506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7702.925397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7704.038615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7711.721263] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7721.057849] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7734.754025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7734.761487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7734.780109] Lustre: Skipped 2 previous similar messages [ 7734.792295] Lustre: Skipped 2 previous similar messages [ 7739.876898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7739.893588] Lustre: Skipped 2 previous similar messages [ 7741.056525] Lustre: server umount lustre-MDT0000 complete [ 7745.002057] LustreError: 357489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.028931] LustreError: 357489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7745.130653] LustreError: 359326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784240429 with bad export cookie 15452488816418419780 [ 7745.132549] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7745.147132] LustreError: 359326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7745.585783] Lustre: server umount lustre-MDT0001 complete [ 7760.235422] Lustre: server umount lustre-OST0000 complete [ 7774.456122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7775.019594] LustreError: 361209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7775.094462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7780.115063] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7780.322606] LustreError: 361210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7785.443829] LustreError: 361209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7790.483943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7790.937428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7796.260108] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7804.885604] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7811.841530] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7818.767923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7819.050572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7820.102724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7825.929858] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7834.803812] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7852.008199] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7852.027161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7852.036818] Lustre: Skipped 1 previous similar message [ 7852.062306] Lustre: Skipped 2 previous similar messages [ 7854.898119] Lustre: server umount lustre-MDT0000 complete [ 7857.121826] LustreError: 361209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7857.165434] LustreError: 361209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7858.589937] LustreError: 361192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784240542 with bad export cookie 15452488816418420886 [ 7858.593164] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7858.601834] LustreError: 361192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7859.104036] Lustre: server umount lustre-MDT0001 complete [ 7873.917212] Lustre: server umount lustre-OST0000 complete [ 7891.630066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7892.164754] LustreError: 365103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7892.247747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7897.034333] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7906.101092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7911.947202] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7921.384985] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7927.727924] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7936.114411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7936.614749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7936.632705] Lustre: Skipped 1 previous similar message [ 7937.657396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7944.330640] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7953.103151] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7958.501360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7958.514105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7958.525794] Lustre: Skipped 1 previous similar message [ 7958.529793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7961.578235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.713514] Lustre: server umount lustre-OST0000 complete [ 7971.808579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7971.812297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7971.828504] Lustre: Skipped 1 previous similar message [ 7971.844221] Lustre: Skipped 3 previous similar messages [ 7984.095185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7984.339880] Lustre: server umount lustre-MDT0000 complete [ 7986.146147] LustreError: 365104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7986.159080] LustreError: 365104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7988.189514] LustreError: 365084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784240672 with bad export cookie 15452488816418421915 [ 7988.193220] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7988.206145] LustreError: 365084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7988.740845] Lustre: server umount lustre-MDT0001 complete [ 7999.808673] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 8002.801864] Key type lgssc unregistered [ 8003.032769] LNet: 368622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8003.042380] LNetError: 368622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8003.062178] LNet: Removed LNI 192.168.204.128@tcp [ 8003.930501] Key type .llcrypt unregistered [ 8003.933079] Key type ._llcrypt unregistered [ 8031.311307] Key type ._llcrypt registered [ 8031.313973] Key type .llcrypt registered [ 8031.405803] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8032.770680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8032.806635] alg: No test for adler32 (adler32-zlib) [ 8033.877680] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 8034.097383] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 8035.775203] Key type lgssc registered [ 8036.945819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8044.392815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8044.404445] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8044.424228] LustreError: 370059:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8044.454608] LustreError: 370059:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8044.467230] LustreError: 370059:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8044.476958] LustreError: 370059:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8044.488733] LustreError: 370059:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8052.740818] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8052.754083] LustreError: 370455:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8052.771876] LustreError: 370455:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8052.780951] LustreError: 370455:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8052.786872] LustreError: 370455:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8052.793757] LustreError: 370455:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8071.505951] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 8074.516395] Key type lgssc unregistered [ 8074.795362] LNet: 371456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8074.802844] LNetError: 371456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8074.825793] LNet: Removed LNI 192.168.204.128@tcp [ 8075.796446] Key type .llcrypt unregistered [ 8075.802218] Key type ._llcrypt unregistered [ 8095.112385] Key type ._llcrypt registered [ 8095.116352] Key type .llcrypt registered [ 8095.240242] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 8113.184095] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8114.981630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8115.092679] alg: No test for adler32 (adler32-zlib) [ 8116.844785] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 8117.323622] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 8118.983181] Key type lgssc registered [ 8120.534941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8161.124392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8161.155026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8162.540903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8162.587523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8162.664568] Lustre: lustre-MDT0000: new disk, initializing [ 8162.737717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8162.753292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8167.876911] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8179.771794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8179.951226] Lustre: 374604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8180.018345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8180.029995] Lustre: Skipped 1 previous similar message [ 8180.149130] Lustre: lustre-MDT0001: new disk, initializing [ 8180.316580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8180.355240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8180.382921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8186.396347] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8198.313391] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8205.356436] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8213.136306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8213.515235] Lustre: lustre-OST0000: new disk, initializing [ 8213.545448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8213.551068] Lustre: 376566:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8213.641583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8220.698654] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8220.747971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8220.765972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8220.938940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8230.668704] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8235.471229] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8235.753420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8241.273853] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8241.515421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8246.754253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.774106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8246.782616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8246.815059] LustreError: Skipped 1 previous similar message [ 8249.968517] Lustre: server umount lustre-OST0000 complete [ 8257.006888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8257.007570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8257.031734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8257.031744] Lustre: Skipped 1 previous similar message [ 8257.068476] Lustre: Skipped 2 previous similar messages [ 8260.923611] Lustre: server umount lustre-MDT0000 complete [ 8264.472889] LustreError: 374596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784240948 with bad export cookie 10626820060829806382 [ 8264.491524] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8264.495240] LustreError: 374596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8265.223377] Lustre: server umount lustre-MDT0001 complete [ 8274.652917] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8276.328520] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:29:19 (1784240959) [ 8283.351444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8283.763863] LustreError: 379016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8283.794512] LustreError: 379016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8283.874364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8288.417306] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8289.253311] LustreError: 379017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.368939] LustreError: 379016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8296.446591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8296.740896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8301.584892] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8308.820907] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8314.925562] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8322.103747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8322.575408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8330.538925] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8339.624196] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8346.457033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8346.567801] Lustre: lustre-OST0001: new disk, initializing [ 8346.571801] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8346.576879] Lustre: 381844:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8346.657111] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8353.833686] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8353.918991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8353.941114] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8354.099515] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8363.888659] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8369.631959] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8369.636054] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8369.648987] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8369.669536] Lustre: Skipped 1 previous similar message [ 8369.687054] Lustre: Skipped 2 previous similar messages [ 8374.219659] Lustre: server umount lustre-OST0001 complete [ 8374.766623] LustreError: 380834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8374.788250] LustreError: 380834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8379.919602] LustreError: 380834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8379.934642] LustreError: 380834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8390.144094] LustreError: 382410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8390.171869] LustreError: 382410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8395.247168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8395.258956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8395.261358] Lustre: Skipped 1 previous similar message [ 8395.282920] Lustre: Skipped 2 previous similar messages [ 8399.614204] Lustre: server umount lustre-OST0000 complete [ 8404.448689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8404.456950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8404.484115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8409.587787] Lustre: server umount lustre-MDT0000 complete [ 8410.598328] LustreError: 382650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8410.627562] LustreError: 382650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8413.741879] LustreError: 380855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784241097 with bad export cookie 10626820060829807264 [ 8413.756913] LustreError: 380855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8413.762841] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8420.247590] Lustre: server umount lustre-MDT0001 complete [ 8430.878363] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 8433.562486] Key type lgssc unregistered [ 8433.869879] LNet: 383822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8433.885024] LNetError: 383822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8434.920214] LNet: Removed LNI 192.168.204.128@tcp [ 8435.872115] Key type .llcrypt unregistered [ 8435.876883] Key type ._llcrypt unregistered [ 8486.158543] Key type ._llcrypt registered [ 8486.161613] Key type .llcrypt registered [ 8486.258727] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8486.906536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8486.958137] alg: No test for adler32 (adler32-zlib) [ 8488.157620] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 8488.424445] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 8490.135161] Key type lgssc registered [ 8491.053443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8498.723354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8498.755432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8500.088342] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8500.124989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8500.359195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8504.369091] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8512.491530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8512.590466] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8512.674490] Lustre: 386612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8512.968112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8517.100405] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8524.413751] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8529.702705] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8533.474098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8533.485840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8533.517551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8533.989457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8537.928263] Lustre: server umount lustre-MDT0000 complete [ 8541.261459] LustreError: 388200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784241225 with bad export cookie 10076656524769567308 [ 8541.264741] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8541.274624] LustreError: 388200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8541.578343] Lustre: server umount lustre-MDT0001 complete [ 8550.638498] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:33:53 (1784241233) [ 8557.990227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8558.355985] LustreError: 389115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8558.372524] LustreError: 389115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8558.412877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8562.711919] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8563.681100] LustreError: 389116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8568.801984] LustreError: 389115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8571.317770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8572.009701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8577.390637] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8584.851673] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8590.285595] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8596.760494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8596.926944] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8597.172205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8603.657671] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8612.146604] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8621.781171] Lustre: Permanently deactivating lustre-OST0000 [ 8621.803167] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8628.753376] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8629.740109] LustreError: 392043:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8629.745854] LustreError: 392043:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8629.754228] LustreError: 392043:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8632.136467] LustreError: 392091:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8632.143380] LustreError: 392091:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8640.467407] Lustre: server umount lustre-OST0000 complete [ 8643.870725] LustreError: 392339:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8643.877729] LustreError: 392339:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8648.672739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8648.680729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8648.689643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8648.692336] Lustre: Skipped 1 previous similar message [ 8651.233222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8651.237141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8656.357918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8656.373703] Lustre: Skipped 2 previous similar messages [ 8657.512506] Lustre: server umount lustre-MDT0000 complete [ 8661.472561] LustreError: 389111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8661.509128] LustreError: 389111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8661.618935] LustreError: 389096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784241345 with bad export cookie 10076656524769567889 [ 8661.637654] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8661.650076] LustreError: 389096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8671.464585] Lustre: server umount lustre-MDT0001 complete [ 8682.351696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.531083] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8691.010352] LustreError: 393492:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8698.772799] Lustre: 393831:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.128@tcp [ 8701.007843] Lustre: server umount MGS complete [ 8713.636305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8714.031578] LustreError: 394553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.053935] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8714.057925] Lustre: Skipped 1 previous similar message [ 8714.099571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8719.233469] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8724.448119] LustreError: 394554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8724.477106] LustreError: 394554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8727.720555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.039316] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8732.165824] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8740.430252] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8746.596548] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8748.886603] Lustre: Permanently reactivating lustre-OST0000 [ 8748.914679] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8748.920463] Lustre: Skipped 2 previous similar messages [ 8775.139666] LustreError: 396281:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8783.746736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8784.035238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8784.040451] Lustre: Skipped 1 previous similar message [ 8790.193835] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8797.025260] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8804.836734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8804.849532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8804.876682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8809.196491] Lustre: server umount lustre-OST0000 complete [ 8815.072290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8815.088732] LustreError: Skipped 1 previous similar message [ 8815.103079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8815.129223] Lustre: Skipped 1 previous similar message [ 8815.144152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8815.155508] Lustre: Skipped 2 previous similar messages [ 8819.615737] Lustre: server umount lustre-MDT0000 complete [ 8821.215881] LustreError: 395792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8821.250032] LustreError: 395792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8824.158232] LustreError: 394533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784241508 with bad export cookie 10076656524769568939 [ 8824.168244] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8824.176095] LustreError: 394533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8824.576521] Lustre: server umount lustre-MDT0001 complete [ 8835.462547] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 8838.059562] Key type lgssc unregistered [ 8838.360871] LNet: 398900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8838.368391] LNetError: 398900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8838.381533] LNet: Removed LNI 192.168.204.128@tcp [ 8839.380960] Key type .llcrypt unregistered [ 8839.388990] Key type ._llcrypt unregistered [ 8857.621568] Key type ._llcrypt registered [ 8857.623875] Key type .llcrypt registered [ 8857.724830] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 8874.362616] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8876.042493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8876.093524] alg: No test for adler32 (adler32-zlib) [ 8877.212614] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 8877.474403] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 8879.128042] Key type lgssc registered [ 8881.889119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8924.620739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8924.650071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8925.888387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8925.983849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8926.089372] Lustre: lustre-MDT0000: new disk, initializing [ 8926.236242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8926.255362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8931.411367] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8941.797035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8941.975497] Lustre: 402048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8942.052319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8942.055257] Lustre: Skipped 1 previous similar message [ 8942.106714] Lustre: lustre-MDT0001: new disk, initializing [ 8942.188950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8942.243790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8942.256581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8946.912991] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8956.524569] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8961.734919] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8970.211631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8970.745449] Lustre: lustre-OST0000: new disk, initializing [ 8970.749418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8970.762492] Lustre: 404010:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8970.885828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8977.052994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8977.065062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8977.267180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8980.794544] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8994.592572] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8999.804527] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9000.050622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9005.106762] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9005.371466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9008.100288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9008.107543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9008.130428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9013.221147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9013.241482] Lustre: Skipped 2 previous similar messages [ 9013.722458] Lustre: server umount lustre-OST0000 complete [ 9018.340912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9018.349300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9018.361212] Lustre: Skipped 2 previous similar messages [ 9018.374941] Lustre: Skipped 1 previous similar message [ 9023.456305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9023.462570] Lustre: Skipped 1 previous similar message [ 9024.278532] Lustre: server umount lustre-MDT0000 complete [ 9028.159285] LustreError: 402840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784241712 with bad export cookie 16648783634945170659 [ 9028.161604] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9028.167020] LustreError: 402840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9028.547959] Lustre: server umount lustre-MDT0001 complete [ 9039.982756] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9042.153781] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:42:04 (1784241724) [ 9050.192476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9050.746038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9056.229199] LustreError: 406463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9056.256826] LustreError: 406463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9056.366671] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9060.335816] LustreError: 406464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9065.442736] LustreError: 406463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9066.595887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9066.908646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9070.719543] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9076.512306] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9080.987455] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9085.522736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9085.774507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9089.387222] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9093.526763] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9094.493670] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9096.361063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9102.325116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9102.328101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9102.340974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9105.376428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9105.391987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9108.112645] Lustre: server umount lustre-OST0000 complete [ 9113.060560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9113.078879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9113.107161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9113.121779] Lustre: Skipped 2 previous similar messages [ 9119.627934] Lustre: server umount lustre-MDT0000 complete [ 9120.740502] LustreError: 406463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9120.753376] LustreError: 406463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9125.573865] LustreError: 406445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784241809 with bad export cookie 16648783634945171541 [ 9125.584321] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9125.586802] LustreError: 406445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9125.870306] Lustre: server umount lustre-MDT0001 complete [ 9140.468708] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 9144.268738] Key type lgssc unregistered [ 9144.704909] LNet: 410174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9144.712430] LNetError: 410174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9144.731449] LNet: Removed LNI 192.168.204.128@tcp [ 9145.831100] Key type .llcrypt unregistered [ 9145.833078] Key type ._llcrypt unregistered [ 9175.694880] Key type ._llcrypt registered [ 9175.701423] Key type .llcrypt registered [ 9175.843895] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 9176.163343] Key type .llcrypt unregistered [ 9176.166688] Key type ._llcrypt unregistered [ 9199.142717] Key type ._llcrypt registered [ 9199.151504] Key type .llcrypt registered [ 9199.314808] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9201.091899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9201.123353] alg: No test for adler32 (adler32-zlib) [ 9202.170276] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 9202.489726] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 9204.231236] Key type lgssc registered [ 9205.742728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9215.225720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9215.256339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9216.967893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9222.945869] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9227.246387] LustreError: 412398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9227.295825] LustreError: 412398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9232.355893] LustreError: 412399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9233.013306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9233.490482] LustreError: 412398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9233.526326] LustreError: 412398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9233.541827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9239.015595] LustreError: 412399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9239.035463] LustreError: 412399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9240.421145] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9242.129136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9247.203812] LustreError: 412399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9247.230160] LustreError: 412399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9249.331112] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9256.416106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9256.645870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9261.780211] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9277.422023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9277.428327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9277.444161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9277.474965] Lustre: Skipped 1 previous similar message [ 9282.531545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9282.552252] Lustre: Skipped 1 previous similar message [ 9287.662243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9287.674890] Lustre: Skipped 1 previous similar message [ 9288.160049] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9288.566623] Lustre: server umount lustre-OST0000 complete [ 9297.889207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9297.890136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9297.891438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9297.922133] Lustre: Skipped 2 previous similar messages [ 9303.009797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9303.018187] Lustre: Skipped 2 previous similar messages [ 9308.128229] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9308.451780] Lustre: server umount lustre-MDT0000 complete [ 9313.253891] LustreError: 412394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9313.281390] LustreError: 412394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9313.429969] LustreError: 412379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784241997 with bad export cookie 3823498766242887881 [ 9313.432204] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9313.453307] LustreError: 412379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9314.057576] Lustre: server umount lustre-MDT0001 complete [ 9329.836251] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 9333.412858] Key type lgssc unregistered [ 9333.764213] LNet: 415534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9333.773917] LNetError: 415534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9333.804960] LNet: Removed LNI 192.168.204.128@tcp [ 9335.046890] Key type .llcrypt unregistered [ 9335.048668] Key type ._llcrypt unregistered [ 9363.405387] Key type ._llcrypt registered [ 9363.411374] Key type .llcrypt registered [ 9363.539203] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9365.003684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9365.052095] alg: No test for adler32 (adler32-zlib) [ 9366.133646] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 9366.387257] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 9368.106100] Key type lgssc registered [ 9369.452417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9378.351562] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9378.382377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9394.080636] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9399.199352] LustreError: 416918:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 9425.887451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9433.623304] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9444.655361] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9455.268099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9455.987070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9456.162544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9460.691733] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9465.319229] LustreError: 417944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9465.332057] LustreError: 417944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9470.433418] LustreError: 416943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9470.474290] LustreError: 416943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9470.830349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9471.541607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9477.377344] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9485.737833] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9491.179566] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9502.208760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9502.209905] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9502.224979] Lustre: Skipped 1 previous similar message [ 9502.226453] LustreError: 416931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9502.486442] Lustre: server umount lustre-OST0000 complete [ 9507.298843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9507.334269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9507.345355] Lustre: Skipped 1 previous similar message [ 9512.421405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9512.429909] Lustre: Skipped 1 previous similar message [ 9512.922766] Lustre: server umount lustre-MDT0000 complete [ 9517.350861] LustreError: 416924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784242201 with bad export cookie 3475366959126683388 [ 9517.356640] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9517.370280] LustreError: 416924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9517.889719] Lustre: server umount lustre-MDT0001 complete [ 9530.184289] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 9533.778388] Key type lgssc unregistered [ 9534.154117] LNet: 420472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9534.173361] LNetError: 420472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9534.196791] LNet: Removed LNI 192.168.204.128@tcp [ 9535.509483] Key type .llcrypt unregistered [ 9535.521090] Key type ._llcrypt unregistered [ 9563.595254] Key type ._llcrypt registered [ 9563.598862] Key type .llcrypt registered [ 9563.784989] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9565.282203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9565.346178] alg: No test for adler32 (adler32-zlib) [ 9566.489611] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 9566.826566] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 9568.511606] Key type lgssc registered [ 9569.970278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9578.173879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9578.210263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9579.763065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9585.111780] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9589.220127] LustreError: 421883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9589.241342] LustreError: 421883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9594.338123] LustreError: 421884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9594.937984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9595.717438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9600.477562] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9609.151363] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9616.370620] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9624.303719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9624.486548] LustreError: 423699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9624.620685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9629.707565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9633.063564] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9641.335798] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9647.072849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9647.080253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9647.102586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9648.188879] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 9650.150710] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9650.153280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9650.175964] Lustre: Skipped 1 previous similar message [ 9650.189430] Lustre: Skipped 2 previous similar messages [ 9652.014436] Lustre: server umount lustre-MDT0000 complete [ 9653.265441] LustreError: 421878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9655.677889] LustreError: 421863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.28@tcp arrived at 1784242339 with bad export cookie 16409776899026082490 [ 9655.702995] LustreError: 421863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9657.789761] LustreError: 424453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784242342 with bad export cookie 16409776899026081587 [ 9657.807254] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9660.385540] LustreError: 422610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9660.386833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9660.424181] LustreError: 422610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9660.476530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9660.494810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9664.176035] Lustre: server umount lustre-OST0000 complete [ 9668.588214] LustreError: 421878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9668.605324] LustreError: 421878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9672.106282] Lustre: server umount lustre-MDT0001 complete [ 9685.923661] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 9689.948579] Key type lgssc unregistered [ 9690.585437] LNet: 425553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9690.599544] LNetError: 425553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9690.625694] LNet: Removed LNI 192.168.204.128@tcp [ 9692.063624] Key type .llcrypt unregistered [ 9692.065567] Key type ._llcrypt unregistered [ 9722.286959] Key type ._llcrypt registered [ 9722.289675] Key type .llcrypt registered [ 9722.447774] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9723.584631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9723.679615] alg: No test for adler32 (adler32-zlib) [ 9724.853916] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 9725.166614] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 9726.911165] Key type lgssc registered [ 9727.998404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9736.825136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9736.868342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9738.597637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9743.508663] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9747.946171] LustreError: 426965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9747.975383] LustreError: 426965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9752.271918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9752.892455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9758.164883] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9767.876251] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9774.429143] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9782.351904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9782.811421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9787.913963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9790.304134] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9798.025853] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9803.242843] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9803.268082] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9803.286605] Lustre: Skipped 1 previous similar message [ 9805.214653] Lustre: server umount lustre-MDT0001 complete [ 9805.886611] LustreError: 426961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.354219] LustreError: 426960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.381135] LustreError: 426960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9813.479531] LustreError: 427946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9813.485642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9813.485899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9813.485904] Lustre: Skipped 1 previous similar message [ 9813.487101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9818.470499] Lustre: server umount lustre-OST0000 complete [ 9818.609150] LustreError: 426966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9823.149786] Lustre: server umount lustre-MDT0000 complete [ 9839.988481] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 9842.924746] Key type lgssc unregistered [ 9843.427319] LNet: 430638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9843.438922] LNetError: 430638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9843.481782] LNet: Removed LNI 192.168.204.128@tcp [ 9844.535290] Key type .llcrypt unregistered [ 9844.538743] Key type ._llcrypt unregistered [ 9873.391405] Key type ._llcrypt registered [ 9873.393747] Key type .llcrypt registered [ 9873.538290] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 9891.460582] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9892.609407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9892.661518] alg: No test for adler32 (adler32-zlib) [ 9894.141668] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [ 9894.515663] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 9896.287232] Key type lgssc registered [ 9897.661783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9938.905662] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9955.398480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9955.416388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9956.730369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9956.763513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9956.909718] Lustre: lustre-MDT0000: new disk, initializing [ 9957.019272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9957.034310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9961.729575] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9977.352559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9977.717415] Lustre: lustre-OST0000: new disk, initializing [ 9977.723262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9977.735210] Lustre: Skipped 1 previous similar message [ 9977.749838] Lustre: 435656:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9977.858312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9983.581878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9983.599077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9983.640039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9985.593649] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9997.891635] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10014.927345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10015.086850] Lustre: 434641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10015.189683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10015.351432] Lustre: lustre-MDT0001: new disk, initializing [10015.520443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10015.563179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10015.576808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10020.083270] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10030.246693] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10030.465686] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10034.964180] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10035.183703] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10044.902496] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10044.905762] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10044.927848] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10045.990307] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [10045.998863] Lustre: Skipped 1 previous similar message [10047.243936] Lustre: server umount lustre-MDT0001 complete [10055.143479] LustreError: 434653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10055.176076] LustreError: 434653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10060.258400] LustreError: 438219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10060.279274] LustreError: 438219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10065.380117] LustreError: 434649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10065.408331] LustreError: 434649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10070.496732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10070.500883] LustreError: 438219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10070.535080] Lustre: Skipped 1 previous similar message [10070.540060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10071.747472] Lustre: server umount lustre-OST0000 complete [10075.638910] LustreError: 434653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10082.306580] Lustre: server umount lustre-MDT0000 complete [10098.194857] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [10101.071449] Key type lgssc unregistered [10101.366525] LNet: 439389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10101.376895] LNetError: 439389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10102.440372] LNet: Removed LNI 192.168.204.128@tcp [10103.373480] Key type .llcrypt unregistered [10103.379300] Key type ._llcrypt unregistered [10151.868447] Key type ._llcrypt registered [10151.870505] Key type .llcrypt registered [10151.986057] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [10152.268600] Key type .llcrypt unregistered [10152.272728] Key type ._llcrypt unregistered [10167.892842] Key type ._llcrypt registered [10167.895169] Key type .llcrypt registered [10167.991151] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [10169.071675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10169.295848] alg: No test for adler32 (adler32-zlib) [10170.389795] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [10170.661318] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [10172.327177] Key type lgssc registered [10173.450877] Lustre: Echo OBD driver; http://www.lustre.org/ [10200.728570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10200.748249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10201.969560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10202.012873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10202.166847] Lustre: lustre-MDT0000: new disk, initializing [10202.254852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10202.274251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10206.732742] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10216.371137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10216.487276] Lustre: 442766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10216.554927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10216.559592] Lustre: Skipped 1 previous similar message [10216.662612] Lustre: lustre-MDT0001: new disk, initializing [10216.760960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10216.824067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10216.843612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10220.664169] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10229.442535] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10235.621890] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10243.487336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10243.938789] Lustre: lustre-OST0000: new disk, initializing [10243.941443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10243.945893] Lustre: 444728:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10244.149315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10251.457842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10251.485180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10251.563084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10251.704252] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10263.056626] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10272.231461] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10272.235247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10272.246255] Lustre: Skipped 2 previous similar messages [10272.259974] Lustre: Skipped 1 previous similar message [10276.778622] Lustre: server umount lustre-MDT0000 complete [10280.804068] LustreError: 442758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784242965 with bad export cookie 11949953764203723977 [10280.808313] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10280.812616] LustreError: 442758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10281.217621] Lustre: server umount lustre-MDT0001 complete [10295.775495] Lustre: server umount lustre-OST0000 complete [10305.658585] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 19:03:08 (1784242988) [10313.119901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10313.623556] LustreError: 446748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10313.639727] LustreError: 446748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10313.684076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10317.416228] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10318.828851] LustreError: 446749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10322.922319] LustreError: 446748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.917688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10326.383592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10330.737398] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10337.855940] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10343.768309] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10364.388660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10364.594278] Lustre: Found index 0 for lustre-OST0000, updating log [10364.804238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10370.800920] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10378.189859] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10384.241573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10394.603729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10394.615822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10394.616036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10394.617680] Lustre: Skipped 1 previous similar message [10395.684807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10395.694870] Lustre: Skipped 2 previous similar messages [10399.916556] Lustre: server umount lustre-OST0000 complete [10418.656725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10418.666320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10418.695254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10421.605789] LNet: 1 local NIs in recovery (showing 1): 192.168.204.128@tcp [10422.881169] Lustre: server umount lustre-MDT0000 complete [10423.329246] LustreError: 446743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10423.347754] LustreError: 446743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10426.317939] LustreError: 446729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784243110 with bad export cookie 11949953764203725370 [10426.319061] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10426.327079] LustreError: 446729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10426.657627] Lustre: server umount lustre-MDT0001 complete [10434.585324] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 19:05:17 (1784243117) [10485.088056] LNet: 1 local NIs in recovery (showing 1): 192.168.204.128@tcp [11059.626535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11060.112822] LustreError: 453078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11060.213103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11064.971786] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11065.315177] LustreError: 453079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11070.436089] LustreError: 453078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11075.728488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11076.148802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11081.522232] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11091.624519] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11098.875821] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11112.800064] LNet: 1 local NIs in recovery (showing 1): 192.168.204.128@tcp [11121.524902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11121.841394] Lustre: Found index 0 for lustre-OST0000, updating log [11122.263117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11129.023504] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11136.917978] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11140.328464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11140.340617] Lustre: Skipped 1 previous similar message [11140.349439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11140.357615] Lustre: Skipped 4 previous similar messages [11145.503396] Lustre: server umount lustre-OST0000 complete [11150.901592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11150.931398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11150.939862] Lustre: Skipped 1 previous similar message [11151.971695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11151.985037] Lustre: Skipped 1 previous similar message [11154.021155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11154.039109] Lustre: Skipped 3 previous similar messages [11156.321657] Lustre: server umount lustre-MDT0000 complete [11157.094609] LustreError: 453078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11157.116461] LustreError: 453078:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11160.905049] LustreError: 453060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784243845 with bad export cookie 11949953764203726455 [11160.906851] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11160.917739] LustreError: 453060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11161.544043] Lustre: server umount lustre-MDT0001 complete [11173.216311] LNet: 1 local NIs in recovery (showing 1): 192.168.204.128@tcp [11194.496407] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [11206.264237] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11242.060847] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 19:18:45 (1784243925) [11258.644504] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [11261.424632] Key type lgssc unregistered [11261.758664] LNet: 459659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11261.768392] LNetError: 459659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11384.680648] LNet: Removed LNI 192.168.204.128@tcp [11385.547267] Key type .llcrypt unregistered [11385.553391] Key type ._llcrypt unregistered [11401.660523] Key type ._llcrypt registered [11401.664306] Key type .llcrypt registered [11401.806841] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11402.990697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11403.170133] alg: No test for adler32 (adler32-zlib) [11404.320399] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [11404.527854] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [11406.143787] Key type lgssc registered [11407.860514] Lustre: Echo OBD driver; http://www.lustre.org/ [11414.130546] LNet: Added LNI 192.168.204.128@tcp100 [8/256/0/180] [11415.191906] LNet: Added LNI 192.168.204.128@tcp101 [8/256/0/180] [11416.212165] LNet: Added LNI 192.168.204.128@tcp102 [8/256/0/180] [11417.664907] LNet: Added LNI 192.168.204.128@tcp103 [8/256/0/180] [11418.718057] LNet: Added LNI 192.168.204.128@tcp104 [8/256/0/180] [11420.245379] LNet: Added LNI 192.168.204.128@tcp105 [8/256/0/180] [11421.317867] LNet: Added LNI 192.168.204.128@tcp106 [8/256/0/180] [11422.443929] LNet: Added LNI 192.168.204.128@tcp107 [8/256/0/180] [11423.312724] LNet: Added LNI 192.168.204.128@tcp108 [8/256/0/180] [11424.244297] LNet: Added LNI 192.168.204.128@tcp109 [8/256/0/180] [11425.259292] LNet: Added LNI 192.168.204.128@tcp110 [8/256/0/180] [11426.426472] LNet: Added LNI 192.168.204.128@tcp111 [8/256/0/180] [11427.627379] LNet: Added LNI 192.168.204.128@tcp112 [8/256/0/180] [11428.711398] LNet: Added LNI 192.168.204.128@tcp113 [8/256/0/180] [11429.764693] LNet: Added LNI 192.168.204.128@tcp114 [8/256/0/180] [11430.656308] LNet: Added LNI 192.168.204.128@tcp115 [8/256/0/180] [11431.463156] LNet: Added LNI 192.168.204.128@tcp116 [8/256/0/180] [11432.318759] LNet: Added LNI 192.168.204.128@tcp117 [8/256/0/180] [11433.181862] LNet: Added LNI 192.168.204.128@tcp118 [8/256/0/180] [11433.940725] LNet: Added LNI 192.168.204.128@tcp119 [8/256/0/180] [11434.728266] LNet: Added LNI 192.168.204.128@tcp120 [8/256/0/180] [11435.625711] LNet: Added LNI 192.168.204.128@tcp121 [8/256/0/180] [11436.520988] LNet: Added LNI 192.168.204.128@tcp122 [8/256/0/180] [11437.474855] LNet: Added LNI 192.168.204.128@tcp123 [8/256/0/180] [11438.356455] LNet: Added LNI 192.168.204.128@tcp124 [8/256/0/180] [11439.155874] LNet: Added LNI 192.168.204.128@tcp125 [8/256/0/180] [11439.958945] LNet: Added LNI 192.168.204.128@tcp126 [8/256/0/180] [11440.938258] LNet: Added LNI 192.168.204.128@tcp127 [8/256/0/180] [11441.677700] LNet: Added LNI 192.168.204.128@tcp128 [8/256/0/180] [11442.484591] LNet: Added LNI 192.168.204.128@tcp129 [8/256/0/180] [11443.157840] LNet: Added LNI 192.168.204.128@tcp130 [8/256/0/180] [11444.003743] LNet: Added LNI 192.168.204.128@tcp131 [8/256/0/180] [11444.828434] LNet: Added LNI 192.168.204.128@tcp132 [8/256/0/180] [11445.806587] LNet: Added LNI 192.168.204.128@tcp133 [8/256/0/180] [11446.591645] LNet: Added LNI 192.168.204.128@tcp134 [8/256/0/180] [11447.409429] LNet: Added LNI 192.168.204.128@tcp135 [8/256/0/180] [11454.629336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11454.674464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11455.867688] Lustre: 462667:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.128@tcp: can't add NID 192.168.204.128@tcp130: rc = -75 [11456.022556] LNetError: 462688:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [11456.041276] LNetError: Protocol error connecting to 192.168.204.128@tcp121 at host 192.168.204.128:988: is it running a compatible version of Lustre? [11456.069344] LNetError: Skipped 11 previous similar messages [11456.140712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11456.191474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11456.289558] Lustre: lustre-MDT0000: new disk, initializing [11456.395423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11456.421892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11456.800131] LNet: 5 local NIs in recovery (showing 5): 192.168.204.128@tcp126, 192.168.204.128@tcp127, 192.168.204.128@tcp129, 192.168.204.128@tcp128, 192.168.204.128@tcp121 [11457.120320] LNetError: 460132:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [11457.153086] LNetError: 460132:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [11457.162304] LNetError: Protocol error connecting to 192.168.204.128@tcp117 at host 192.168.204.128:988: is it running a compatible version of Lustre? [11459.940906] LNetError: 462694:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [11459.964719] LNetError: 462694:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [11459.985662] LNetError: Protocol error connecting to 192.168.204.128@tcp127 at host 192.168.204.128:988: is it running a compatible version of Lustre? [11460.002772] LNetError: Skipped 5 previous similar messages [11462.311962] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11471.200532] LNetError: 460134:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [11471.218788] LNetError: Protocol error connecting to 192.168.204.128@tcp127 at host 192.168.204.128:988: is it running a compatible version of Lustre? [11474.608506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11474.746375] Lustre: 462683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11474.850299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11474.857428] Lustre: Skipped 1 previous similar message [11474.985880] Lustre: lustre-MDT0001: new disk, initializing [11475.039251] Lustre: 463485:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.128@tcp: can't add NID 192.168.204.128@tcp130: rc = -75 [11475.063270] Lustre: 463485:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11475.230658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11475.290652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11475.303833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11479.834967] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11485.220652] Lustre: 463933:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.128@tcp: can't add NID 192.168.204.128@tcp130: rc = -75 [11485.237158] Lustre: 463933:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11486.438786] LNetError: 462691:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [11486.449570] LNetError: Protocol error connecting to 192.168.204.128@tcp118 at host 192.168.204.128:988: is it running a compatible version of Lustre? [11489.136680] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11494.676943] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11495.659740] LNetError: 460134:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [11495.670543] LNetError: 460134:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [11495.697562] LNetError: Protocol error connecting to 192.168.204.128@tcp112 at host 192.168.204.128:988: is it running a compatible version of Lustre? [11495.721181] LNetError: Skipped 6 previous similar messages [11501.165203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11501.411610] Lustre: lustre-OST0000: new disk, initializing [11501.416814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11501.428570] Lustre: 464653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11501.547532] Lustre: 464658:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.128@tcp: can't add NID 192.168.204.128@tcp130: rc = -75 [11501.555755] Lustre: 464658:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [11501.617837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11507.068742] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11511.350918] Lustre: 465171:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.128@tcp: can't add NID 192.168.204.128@tcp130: rc = -75 [11511.359710] Lustre: 465171:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11511.419143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11511.434439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11511.524056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11512.035902] LNetError: 462691:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [11512.056652] LNetError: 462691:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [11512.065174] LNetError: Protocol error connecting to 192.168.204.128@tcp113 at host 192.168.204.128:988: is it running a compatible version of Lustre? [11512.074848] LNetError: Skipped 10 previous similar messages [11515.637588] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11517.282295] LNet: 1 local NIs in recovery (showing 1): 192.168.204.128@tcp127 [11527.137100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11527.139216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11527.149589] LustreError: Skipped 1 previous similar message [11527.164787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11527.175054] Lustre: Skipped 1 previous similar message [11528.787913] Lustre: server umount lustre-OST0000 complete [11536.866866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11536.874587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11536.886861] Lustre: Skipped 1 previous similar message [11536.892089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11538.898849] Lustre: server umount lustre-MDT0000 complete [11542.874755] LustreError: 462676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784244227 with bad export cookie 9602104832169960329 [11542.886974] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11542.888343] LustreError: 462676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11543.220637] Lustre: server umount lustre-MDT0001 complete [11545.831761] LNetError: 462688:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.128 [11545.842920] LNetError: 462688:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [11545.851066] LNetError: Protocol error connecting to 192.168.204.128@tcp117 at host 192.168.204.128:988: is it running a compatible version of Lustre? [11545.864096] LNetError: Skipped 13 previous similar messages [11554.748635] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [11557.365670] Key type lgssc unregistered [11557.794247] LNet: 466559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11557.808596] LNetError: 466559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11557.816681] LNet: Removed LNI 192.168.204.128@tcp [11558.880835] LNet: Removed LNI 192.168.204.128@tcp100 [11558.910921] LNetError: 466559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11558.921328] LNetError: 466559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11558.934422] LNet: Removed LNI 192.168.204.128@tcp101 [11559.967250] LNet: Removed LNI 192.168.204.128@tcp102 [11559.972192] LNetError: 466559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11559.988840] LNetError: 466559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11560.002540] LNet: Removed LNI 192.168.204.128@tcp103 [11561.055235] LNet: Removed LNI 192.168.204.128@tcp104 [11562.080992] LNet: Removed LNI 192.168.204.128@tcp105 [11562.087531] LNetError: 466559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11562.105529] LNetError: 466559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11562.121855] LNet: Removed LNI 192.168.204.128@tcp106 [11563.167718] LNet: Removed LNI 192.168.204.128@tcp107 [11564.191246] LNet: Removed LNI 192.168.204.128@tcp108 [11565.215325] LNet: Removed LNI 192.168.204.128@tcp109 [11566.239194] LNet: Removed LNI 192.168.204.128@tcp110 [11566.246857] LNetError: 466559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11566.257959] LNetError: 466559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11566.276382] LNet: Removed LNI 192.168.204.128@tcp111 [11567.328021] LNet: Removed LNI 192.168.204.128@tcp112 [11568.351181] LNet: Removed LNI 192.168.204.128@tcp113 [11569.375368] LNet: Removed LNI 192.168.204.128@tcp114 [11570.399146] LNet: Removed LNI 192.168.204.128@tcp115 [11571.423194] LNet: Removed LNI 192.168.204.128@tcp116 [11572.448021] LNet: Removed LNI 192.168.204.128@tcp117 [11573.471184] LNet: Removed LNI 192.168.204.128@tcp118 [11574.495175] LNet: Removed LNI 192.168.204.128@tcp119 [11574.503483] LNetError: 466559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11574.522623] LNetError: 466559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11574.533667] LNet: Removed LNI 192.168.204.128@tcp120 [11575.583218] LNet: Removed LNI 192.168.204.128@tcp121 [11576.607375] LNet: Removed LNI 192.168.204.128@tcp122 [11577.631197] LNet: Removed LNI 192.168.204.128@tcp123 [11578.655353] LNet: Removed LNI 192.168.204.128@tcp124 [11579.680568] LNet: Removed LNI 192.168.204.128@tcp125 [11580.703254] LNet: Removed LNI 192.168.204.128@tcp126 [11580.705869] LNet: Removed LNI 192.168.204.128@tcp127 [11581.727644] LNet: Removed LNI 192.168.204.128@tcp128 [11582.752348] LNet: Removed LNI 192.168.204.128@tcp129 [11582.761461] LNet: Removed LNI 192.168.204.128@tcp130 [11582.766118] LNet: Removed LNI 192.168.204.128@tcp131 [11582.773371] LNet: Removed LNI 192.168.204.128@tcp132 [11582.780208] LNet: Removed LNI 192.168.204.128@tcp133 [11582.786621] LNet: Removed LNI 192.168.204.128@tcp134 [11582.802721] LNet: Removed LNI 192.168.204.128@tcp135 [11583.623818] Key type .llcrypt unregistered [11583.626040] Key type ._llcrypt unregistered [11607.793374] Key type ._llcrypt registered [11607.796978] Key type .llcrypt registered [11607.949646] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [11622.936507] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11624.243694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11624.262286] alg: No test for adler32 (adler32-zlib) [11625.296040] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [11625.657122] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [11627.343206] Key type lgssc registered [11628.932297] Lustre: Echo OBD driver; http://www.lustre.org/ [11663.950187] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 19:25:47 (1784244347) [11681.741555] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [11684.258852] Key type lgssc unregistered [11684.622507] LNet: 470630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11684.640414] LNetError: 470630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11684.667410] LNet: Removed LNI 192.168.204.128@tcp [11685.822842] Key type .llcrypt unregistered [11685.836229] Key type ._llcrypt unregistered [11704.675482] Key type ._llcrypt registered [11704.677823] Key type .llcrypt registered [11704.826605] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11706.622986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11706.682480] alg: No test for adler32 (adler32-zlib) [11707.938460] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [11708.312804] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [11710.089973] Key type lgssc registered [11711.687307] Lustre: Echo OBD driver; http://www.lustre.org/ [11723.806891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11723.838889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11725.122680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11725.250385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11725.378664] Lustre: lustre-MDT0000: new disk, initializing [11725.491879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11725.523553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11730.508700] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11742.535066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11742.685482] Lustre: 471841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11742.736643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11742.740660] Lustre: Skipped 1 previous similar message [11742.854313] Lustre: lustre-MDT0001: new disk, initializing [11742.988815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11743.064657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11743.083149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11747.697065] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11757.561844] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11764.745359] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11774.746781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11775.265960] Lustre: lustre-OST0000: new disk, initializing [11775.272380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11775.283320] Lustre: 473803:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11775.582420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11784.322107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11784.339879] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11784.529365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11784.681874] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11796.717890] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11801.883105] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing lnet_if_list [11808.579984] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11808.615950] LNet: Added LNI 192.168.204.128@tcp10 [8/256/0/180] [11815.466736] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11815.493475] LNet: Added LNI 192.168.204.128@tcp11 [8/256/0/180] [11820.203564] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11820.228460] LNet: Added LNI 192.168.204.128@tcp12 [8/256/0/180] [11824.813505] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11824.850736] LNet: Added LNI 192.168.204.128@tcp13 [8/256/0/180] [11829.709340] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11829.773195] LNet: Added LNI 192.168.204.128@tcp14 [8/256/0/180] [11835.047738] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11835.078803] LNet: Added LNI 192.168.204.128@tcp15 [8/256/0/180] [11840.091909] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11840.131845] LNet: Added LNI 192.168.204.128@tcp16 [8/256/0/180] [11844.851962] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11844.891451] LNet: Added LNI 192.168.204.128@tcp17 [8/256/0/180] [11849.653798] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11849.688981] LNet: Added LNI 192.168.204.128@tcp18 [8/256/0/180] [11854.922962] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11854.965434] LNet: Added LNI 192.168.204.128@tcp19 [8/256/0/180] [11860.425547] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11860.468806] LNet: Added LNI 192.168.204.128@tcp20 [8/256/0/180] [11866.112121] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11866.167806] LNet: Added LNI 192.168.204.128@tcp21 [8/256/0/180] [11872.334182] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11872.367565] LNet: Added LNI 192.168.204.128@tcp22 [8/256/0/180] [11877.954953] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11877.978408] LNet: Added LNI 192.168.204.128@tcp23 [8/256/0/180] [11884.385460] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11884.448189] LNet: Added LNI 192.168.204.128@tcp24 [8/256/0/180] [11891.337824] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11891.388386] LNet: Added LNI 192.168.204.128@tcp25 [8/256/0/180] [11898.236908] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11898.271452] LNet: Added LNI 192.168.204.128@tcp26 [8/256/0/180] [11905.315693] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11905.369687] LNet: Added LNI 192.168.204.128@tcp27 [8/256/0/180] [11912.681351] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11912.711331] LNet: Added LNI 192.168.204.128@tcp28 [8/256/0/180] [11919.817826] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11919.866365] LNet: Added LNI 192.168.204.128@tcp29 [8/256/0/180] [11927.333310] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11927.385419] LNet: Added LNI 192.168.204.128@tcp30 [8/256/0/180] [11934.941856] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11935.010267] LNet: Added LNI 192.168.204.128@tcp31 [8/256/0/180] [11942.267269] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11942.306231] LNet: Added LNI 192.168.204.128@tcp32 [8/256/0/180] [11949.201794] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11949.230671] LNet: Added LNI 192.168.204.128@tcp33 [8/256/0/180] [11956.975882] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11957.004743] LNet: Added LNI 192.168.204.128@tcp34 [8/256/0/180] [11964.425261] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11964.456443] LNet: Added LNI 192.168.204.128@tcp35 [8/256/0/180] [11971.707717] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11971.765866] LNet: Added LNI 192.168.204.128@tcp36 [8/256/0/180] [11979.569110] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11979.590586] LNet: Added LNI 192.168.204.128@tcp37 [8/256/0/180] [11987.769557] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11987.832344] LNet: Added LNI 192.168.204.128@tcp38 [8/256/0/180] [11997.248582] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11997.293949] LNet: Added LNI 192.168.204.128@tcp39 [8/256/0/180] [12007.536529] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12007.566569] LNet: Added LNI 192.168.204.128@tcp40 [8/256/0/180] [12018.050537] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12018.110356] LNet: Added LNI 192.168.204.128@tcp41 [8/256/0/180] [12028.554033] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12028.593809] LNet: Added LNI 192.168.204.128@tcp42 [8/256/0/180] [12037.711264] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12037.741212] LNet: Added LNI 192.168.204.128@tcp43 [8/256/0/180] [12045.880356] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12045.904890] LNet: Added LNI 192.168.204.128@tcp44 [8/256/0/180] [12054.479549] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12054.502529] LNet: Added LNI 192.168.204.128@tcp45 [8/256/0/180] [12063.013865] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12063.072893] LNet: Added LNI 192.168.204.128@tcp46 [8/256/0/180] [12071.432524] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12071.470332] LNet: Added LNI 192.168.204.128@tcp47 [8/256/0/180] [12079.746564] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12079.779442] LNet: Added LNI 192.168.204.128@tcp48 [8/256/0/180] [12087.933462] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12087.994264] LNet: Added LNI 192.168.204.128@tcp49 [8/256/0/180] [12096.676588] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12096.734199] LNet: Added LNI 192.168.204.128@tcp50 [8/256/0/180] [12116.965140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12116.976831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12116.994950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12117.479081] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12117.502430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12117.521215] Lustre: Skipped 2 previous similar messages [12121.394716] Lustre: server umount lustre-MDT0000 complete [12124.857315] LustreError: 471831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784244809 with bad export cookie 11347820814804681869 [12124.861201] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12124.873348] LustreError: 471831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12125.208593] Lustre: server umount lustre-MDT0001 complete [12142.879981] Lustre: 471112:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784244811/real 1784244811] req@0000000097d17899 x1870915847758720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784244827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12142.908713] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12142.920944] Lustre: Skipped 1 previous similar message [12143.401840] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12146.145386] Lustre: 471114:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784244814/real 1784244814] req@00000000506705cd x1870915847758848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784244830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12149.279243] Lustre: 471114:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784244817/real 1784244817] req@0000000018849734 x1870915847758976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784244833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12152.406874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12156.383214] Lustre: 471113:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784244824/real 1784244824] req@000000000188ffb7 x1870915847759360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784244840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12156.440957] Lustre: 471113:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12170.994995] LustreError: 473796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12171.016441] LustreError: 473796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12171.042441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12172.064641] LustreError: 473797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12172.069118] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12172.088646] LustreError: 473797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12173.094059] LustreError: 473796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12176.204224] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12178.402100] LustreError: 473797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12183.534203] LustreError: 473797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12187.176521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12187.545787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12188.603856] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12193.327532] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12203.649338] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12210.925434] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12233.702601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12233.708302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12233.719543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12239.615588] Lustre: server umount lustre-OST0000 complete [12244.960342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12244.962200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12244.983526] Lustre: Skipped 2 previous similar messages [12245.011807] Lustre: Skipped 3 previous similar messages [12250.081588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12250.089265] Lustre: Skipped 1 previous similar message [12251.368150] Lustre: server umount lustre-MDT0000 complete [12255.200522] LustreError: 497987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12255.231670] LustreError: 497987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12255.728741] LustreError: 473802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784244939 with bad export cookie 11347820814804684347 [12255.745491] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12256.409193] Lustre: server umount lustre-MDT0001 complete [12270.362923] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [12273.554349] Key type lgssc unregistered [12273.939728] LNet: 500777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12273.951191] LNetError: 500777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12274.976047] LNet: Removed LNI 192.168.204.128@tcp [12274.983568] LNetError: 500777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12274.998838] LNet: Removed LNI 192.168.204.128@tcp10 [12275.001525] LNet: Removed LNI 192.168.204.128@tcp11 [12275.012296] LNet: Removed LNI 192.168.204.128@tcp12 [12275.023759] LNet: Removed LNI 192.168.204.128@tcp13 [12275.026990] LNet: Removed LNI 192.168.204.128@tcp14 [12275.038787] LNet: Removed LNI 192.168.204.128@tcp15 [12275.049792] LNet: Removed LNI 192.168.204.128@tcp16 [12275.063322] LNet: Removed LNI 192.168.204.128@tcp17 [12275.072393] LNet: Removed LNI 192.168.204.128@tcp18 [12275.075336] LNet: Removed LNI 192.168.204.128@tcp19 [12275.077275] LNet: Removed LNI 192.168.204.128@tcp20 [12275.082225] LNet: Removed LNI 192.168.204.128@tcp21 [12275.088577] LNet: Removed LNI 192.168.204.128@tcp22 [12275.092203] LNet: Removed LNI 192.168.204.128@tcp23 [12275.103930] LNet: Removed LNI 192.168.204.128@tcp24 [12275.110738] LNet: Removed LNI 192.168.204.128@tcp25 [12275.119700] LNet: Removed LNI 192.168.204.128@tcp26 [12275.125974] LNet: Removed LNI 192.168.204.128@tcp27 [12275.132305] LNet: Removed LNI 192.168.204.128@tcp28 [12275.137376] LNet: Removed LNI 192.168.204.128@tcp29 [12275.140803] LNet: Removed LNI 192.168.204.128@tcp30 [12275.155621] LNet: Removed LNI 192.168.204.128@tcp31 [12275.164570] LNet: Removed LNI 192.168.204.128@tcp32 [12275.171289] LNet: Removed LNI 192.168.204.128@tcp33 [12275.176532] LNet: Removed LNI 192.168.204.128@tcp34 [12275.178776] LNet: Removed LNI 192.168.204.128@tcp35 [12275.190341] LNet: Removed LNI 192.168.204.128@tcp36 [12275.194272] LNet: Removed LNI 192.168.204.128@tcp37 [12275.196312] LNet: Removed LNI 192.168.204.128@tcp38 [12275.198425] LNet: Removed LNI 192.168.204.128@tcp39 [12275.204096] LNet: Removed LNI 192.168.204.128@tcp40 [12275.208307] LNet: Removed LNI 192.168.204.128@tcp41 [12275.211587] LNet: Removed LNI 192.168.204.128@tcp42 [12275.213141] LNet: Removed LNI 192.168.204.128@tcp43 [12275.220941] LNet: Removed LNI 192.168.204.128@tcp44 [12275.231331] LNet: Removed LNI 192.168.204.128@tcp45 [12275.238992] LNet: Removed LNI 192.168.204.128@tcp46 [12275.250496] LNet: Removed LNI 192.168.204.128@tcp47 [12275.252505] LNet: Removed LNI 192.168.204.128@tcp48 [12275.263508] LNet: Removed LNI 192.168.204.128@tcp49 [12275.276278] LNet: Removed LNI 192.168.204.128@tcp50 [12276.188523] Key type .llcrypt unregistered [12276.190920] Key type ._llcrypt unregistered [12300.606679] Key type ._llcrypt registered [12300.614067] Key type .llcrypt registered [12300.843358] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [12316.534662] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12317.726875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12317.760725] alg: No test for adler32 (adler32-zlib) [12318.839566] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [12319.068290] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [12320.736760] Key type lgssc registered [12321.990706] Lustre: Echo OBD driver; http://www.lustre.org/ [12364.428580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12364.499398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12366.103367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12366.173476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12366.381809] Lustre: lustre-MDT0000: new disk, initializing [12366.531563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12366.581475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12372.643257] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12384.196851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12384.318698] Lustre: 503963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12384.378939] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12384.383941] Lustre: Skipped 1 previous similar message [12384.460612] Lustre: lustre-MDT0001: new disk, initializing [12384.544982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12384.579034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12384.600961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12389.820862] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12398.766705] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12403.652217] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12412.724047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12412.942219] Lustre: *** cfs_fail_loc=90f, val=0*** [12413.049139] Lustre: lustre-OST0000: new disk, initializing [12413.058837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12413.074082] Lustre: 505972:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12413.184689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12421.713711] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12422.814135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12422.832637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12423.074443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12434.527684] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12443.617725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12443.643994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12443.647094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12443.673535] Lustre: Skipped 1 previous similar message [12443.702401] Lustre: Skipped 1 previous similar message [12448.736902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12448.762602] Lustre: Skipped 1 previous similar message [12449.199627] Lustre: server umount lustre-OST0000 complete [12453.871306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12453.885959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12453.906806] Lustre: Skipped 1 previous similar message [12458.977287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12458.994213] Lustre: Skipped 2 previous similar messages [12459.488695] Lustre: server umount lustre-MDT0000 complete [12463.380722] LustreError: 503955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784245147 with bad export cookie 8659432139613706662 [12463.386589] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12463.390601] LustreError: 503955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12463.723941] Lustre: server umount lustre-MDT0001 complete [12475.242082] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [12478.314570] Key type lgssc unregistered [12478.586862] LNet: 507871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12478.604793] LNetError: 507871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12478.624639] LNet: Removed LNI 192.168.204.128@tcp [12479.702085] Key type .llcrypt unregistered [12479.704753] Key type ._llcrypt unregistered [12503.510102] Key type ._llcrypt registered [12503.511916] Key type .llcrypt registered [12503.707477] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12505.532173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12505.568612] alg: No test for adler32 (adler32-zlib) [12506.647240] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [12506.931033] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [12508.663588] Key type lgssc registered [12509.745380] Lustre: Echo OBD driver; http://www.lustre.org/ [12517.123946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12517.145570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12518.545703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12522.955934] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12528.106201] LustreError: 509139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12528.118409] LustreError: 509139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12530.868036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12531.225691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12535.730327] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12542.870642] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12548.078750] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12555.260113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12555.640297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12562.474454] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12571.161162] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12582.638114] loop0: detected capacity change from 0 to 204800000 [12582.880101] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12582.888831] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12582.908075] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12590.369683] loop0: detected capacity change from 0 to 204800000 [12590.655996] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12590.693921] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12590.726763] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12597.070357] loop0: detected capacity change from 0 to 204800000 [12602.857979] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12602.972851] Lustre: fs15246-OST0000: new disk, initializing [12602.979710] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12602.987032] Lustre: 512720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12603.051595] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12609.703593] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12619.960274] loop1: detected capacity change from 0 to 204800000 [12625.100394] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12625.203812] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12625.265720] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12625.381135] Lustre: fs15246-MDT0000: new disk, initializing [12625.563117] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12625.602550] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12625.734637] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12626.792931] Lustre: 513838:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000a030a560 x1870916685447552/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784245327 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12629.936086] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12635.948621] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12635.960114] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12635.994228] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12655.080406] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12655.097630] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12657.921219] Lustre: server umount fs15246-MDT0000 complete [12660.201777] LustreError: 509134:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12665.320279] LustreError: 511622:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12666.766975] Lustre: server umount fs15246-OST0000 complete [12679.656878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12679.667117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12679.695392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12680.674717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12680.674970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12680.712685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12680.721475] Lustre: Skipped 1 previous similar message [12685.631721] Lustre: server umount lustre-OST0000 complete [12689.890443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12689.900829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12689.915664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12695.746610] Lustre: server umount lustre-MDT0000 complete [12696.032454] LustreError: 509139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12696.053730] LustreError: 509139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12699.259135] LustreError: 509118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784245383 with bad export cookie 12487688036997626684 [12699.260247] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12699.271695] LustreError: 509118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12699.583883] Lustre: server umount lustre-MDT0001 complete [12711.145623] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [12714.634746] Key type lgssc unregistered [12714.935525] LNet: 516923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12714.947049] LNetError: 516923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12714.980649] LNet: Removed LNI 192.168.204.128@tcp [12716.099208] Key type .llcrypt unregistered [12716.102758] Key type ._llcrypt unregistered [12738.528657] Key type ._llcrypt registered [12738.534219] Key type .llcrypt registered [12738.683801] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12739.880202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12740.001080] alg: No test for adler32 (adler32-zlib) [12741.184572] Lustre: Lustre: Build Version: 2.17.54_162_gb342f71 [12741.496679] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [12743.218068] Key type lgssc registered [12744.355636] Lustre: Echo OBD driver; http://www.lustre.org/ [12786.125524] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [12797.391103] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12836.402570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12836.434128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12837.687375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12837.751571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12837.838138] Lustre: lustre-MDT0000: new disk, initializing [12837.925084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12837.941510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12842.239865] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12853.788765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12853.944066] Lustre: 521377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12854.019078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12854.023847] Lustre: Skipped 1 previous similar message [12854.164292] Lustre: lustre-MDT0001: new disk, initializing [12854.292365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12854.332319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12854.342681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12859.555162] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12869.585248] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12875.482272] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12882.289775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12882.635488] Lustre: lustre-OST0000: new disk, initializing [12882.639560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12882.644616] Lustre: 523336:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12882.802451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12889.174212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12889.196949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12889.317859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12889.828861] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12900.067188] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12905.486687] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12905.702938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12910.839619] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12911.056487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12915.177674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12915.183149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12915.199324] LustreError: Skipped 1 previous similar message [12915.213425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12919.292347] Lustre: server umount lustre-OST0000 complete [12925.410026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12925.433913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12925.435762] Lustre: Skipped 2 previous similar messages [12925.448797] Lustre: Skipped 1 previous similar message [12930.530368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12935.653221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12935.666205] Lustre: Skipped 2 previous similar messages [12938.207571] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12938.519991] Lustre: server umount lustre-MDT0000 complete [12942.865920] LustreError: 521369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784245627 with bad export cookie 13041887724986994054 [12942.874152] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12942.879328] LustreError: 521369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12943.306043] Lustre: server umount lustre-MDT0001 complete [12968.140386] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [12977.901752] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [13008.813392] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12833 sec ========= 19:48:11 (1784245691) [13010.974572] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:48:13 (1784245693) === [13015.196873] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:48:18 (1784245698) === [13060.990131] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [13064.078792] Key type lgssc unregistered [13064.449892] LNet: 529961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13064.466420] LNetError: 529961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13064.485980] LNet: Removed LNI 192.168.204.128@tcp [13065.454124] Key type .llcrypt unregistered [13065.456186] Key type ._llcrypt unregistered