[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 423681792 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003117] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.007945] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.012019] Yama: becoming mindful. [ 0.013024] SELinux: Initializing. [ 0.014057] *** VALIDATE selinux *** [ 0.021502] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026149] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027099] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028066] *** VALIDATE tmpfs *** [ 0.029332] *** VALIDATE proc *** [ 0.030133] *** VALIDATE cgroup *** [ 0.030761] *** VALIDATE cgroup2 *** [ 0.031165] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032087] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034015] Spectre V2 : User space: Vulnerable [ 0.035003] Speculative Store Bypass: Vulnerable [ 0.037505] debug: unmapping init [mem 0xffffffff9da59000-0xffffffff9da60fff] [ 0.040000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040388] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041015] ... version: 2 [ 0.042006] ... bit width: 48 [ 0.043005] ... generic registers: 4 [ 0.043994] ... value mask: 0000ffffffffffff [ 0.044006] ... max period: 00007fffffffffff [ 0.045005] ... fixed-purpose events: 3 [ 0.045888] ... event mask: 000000070000000f [ 0.046202] rcu: Hierarchical SRCU implementation. [ 0.048107] smp: Bringing up secondary CPUs ... [ 0.049339] x86: Booting SMP configuration: [ 0.050014] .... node #0, CPUs: #1 #2 #3 [ 0.054690] smp: Brought up 1 node, 4 CPUs [ 0.056007] smpboot: Max logical packages: 1 [ 0.056843] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.226029] node 0 deferred pages initialised in 169ms [ 0.229354] devtmpfs: initialized [ 0.230265] x86/mm: Memory block size: 128MB [ 0.232034] gcov: version magic: 0x41383552 [ 0.235262] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.239083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.241289] pinctrl core: initialized pinctrl subsystem [ 0.243132] [ 0.243644] ************************************************************* [ 0.246012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.248009] ** ** [ 0.250012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.252012] ** ** [ 0.254011] ** This means that this kernel is built to expose internal ** [ 0.256010] ** IOMMU data structures, which may compromise security on ** [ 0.258013] ** your system. ** [ 0.260011] ** ** [ 0.262010] ** If you see this message and you are not debugging the ** [ 0.265011] ** kernel, report this immediately to your vendor! ** [ 0.267009] ** ** [ 0.269011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.271011] ************************************************************* [ 0.273809] NET: Registered protocol family 16 [ 0.275404] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.278046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.280050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.284046] cpuidle: using governor menu [ 0.285983] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.288675] PCI: Using configuration type 1 for base access [ 0.291129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.300078] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.302016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.305193] cryptd: max_cpu_qlen set to 1000 [ 0.308255] ACPI: Added _OSI(Module Device) [ 0.309008] ACPI: Added _OSI(Processor Device) [ 0.311012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.312007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.315000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.320588] ACPI: Interpreter enabled [ 0.321000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.322007] ACPI: Using IOAPIC for interrupt routing [ 0.324084] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.326365] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.334770] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.337024] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.338011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.341059] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.344235] acpiphp: Slot [2] registered [ 0.346088] acpiphp: Slot [5] registered [ 0.347079] acpiphp: Slot [6] registered [ 0.348087] acpiphp: Slot [7] registered [ 0.349071] acpiphp: Slot [8] registered [ 0.350077] acpiphp: Slot [9] registered [ 0.351070] acpiphp: Slot [10] registered [ 0.352055] acpiphp: Slot [3] registered [ 0.353053] acpiphp: Slot [4] registered [ 0.353962] acpiphp: Slot [11] registered [ 0.355082] acpiphp: Slot [12] registered [ 0.356055] acpiphp: Slot [13] registered [ 0.357052] acpiphp: Slot [14] registered [ 0.358052] acpiphp: Slot [15] registered [ 0.359050] acpiphp: Slot [16] registered [ 0.360051] acpiphp: Slot [17] registered [ 0.361085] acpiphp: Slot [18] registered [ 0.362108] acpiphp: Slot [19] registered [ 0.364090] acpiphp: Slot [20] registered [ 0.365095] acpiphp: Slot [21] registered [ 0.366135] acpiphp: Slot [22] registered [ 0.368092] acpiphp: Slot [23] registered [ 0.369072] acpiphp: Slot [24] registered [ 0.370069] acpiphp: Slot [25] registered [ 0.371070] acpiphp: Slot [26] registered [ 0.373062] acpiphp: Slot [27] registered [ 0.374113] acpiphp: Slot [28] registered [ 0.376122] acpiphp: Slot [29] registered [ 0.377108] acpiphp: Slot [30] registered [ 0.379093] acpiphp: Slot [31] registered [ 0.381078] PCI host bridge to bus 0000:00 [ 0.382014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.385022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.387018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.389054] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.394072] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.399090] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.402274] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.407666] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.411304] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.419016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.423581] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.426011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.428011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.430010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.433573] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.436593] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.438023] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.439642] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.445012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.458000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.462013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.467616] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.479033] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.483012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.498013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.506070] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.511009] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.517014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.529985] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.538816] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.543015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.549021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.564019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.570892] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.576012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.581015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.595017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.603167] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.614015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.625015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.631123] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.636018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.641013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.658022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.679502] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.685369] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.687383] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.690362] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.693256] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.698042] iommu: Default domain type: Passthrough [ 0.699761] SCSI subsystem initialized [ 0.702175] ACPI: bus type USB registered [ 0.704157] usbcore: registered new interface driver usbfs [ 0.707194] usbcore: registered new interface driver hub [ 0.708147] usbcore: registered new device driver usb [ 0.710390] pps_core: LinuxPPS API ver. 1 registered [ 0.713019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.718074] PTP clock support registered [ 0.720130] EDAC MC: Ver: 3.0.0 [ 0.723147] PCI: Using ACPI for IRQ routing [ 0.726040] NetLabel: Initializing [ 0.728013] NetLabel: domain hash size = 128 [ 0.731012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.736113] NetLabel: unlabeled traffic allowed by default [ 0.739042] vgaarb: loaded [ 0.741228] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.748520] clocksource: Switched to clocksource kvm-clock [ 0.848623] VFS: Disk quotas dquot_6.6.0 [ 0.849775] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.851574] *** VALIDATE ramfs *** [ 0.852227] *** VALIDATE hugetlbfs *** [ 0.853171] pnp: PnP ACPI init [ 0.854918] pnp: PnP ACPI: found 6 devices [ 0.867579] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.869715] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.871031] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.873012] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.875099] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.877431] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.880136] NET: Registered protocol family 2 [ 0.882298] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.886247] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.889352] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.893373] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.895687] TCP: Hash tables configured (established 65536 bind 65536) [ 0.898200] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.900905] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.903084] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.905915] NET: Registered protocol family 1 [ 0.908200] RPC: Registered named UNIX socket transport module. [ 0.910682] RPC: Registered udp transport module. [ 0.912101] RPC: Registered tcp transport module. [ 0.913607] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.915631] NET: Registered protocol family 44 [ 0.916965] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.918726] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.920470] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.922386] PCI: CLS 0 bytes, default 64 [ 0.923777] Unpacking initramfs... [ 2.285892] debug: unmapping init [mem 0xffff99453cc54000-0xffff99453ffbffff] [ 2.290323] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.292720] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.296220] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.909814] Initialise system trusted keyrings [ 2.911558] Key type blacklist registered [ 2.913824] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.926198] zbud: loaded [ 2.929214] *** VALIDATE nfs *** [ 2.930493] *** VALIDATE nfs4 *** [ 2.933179] pstore: using deflate compression [ 2.937215] Platform Keyring initialized [ 3.054811] NET: Registered protocol family 38 [ 3.060486] Key type asymmetric registered [ 3.062413] Asymmetric key parser 'x509' registered [ 3.064339] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.067653] io scheduler mq-deadline registered [ 3.069505] io scheduler kyber registered [ 3.071170] io scheduler bfq registered [ 3.074775] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.080258] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.083117] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.085523] ACPI: Power Button [PWRF] [ 3.090777] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.098350] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.113165] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.130578] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.163599] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.193599] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.238974] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.247556] Non-volatile memory driver v1.3 [ 3.251195] Linux agpgart interface v0.103 [ 3.290705] virtio_blk virtio1: [vda] 135992 512-byte logical blocks (69.6 MB/66.4 MiB) [ 3.295140] vda: detected capacity change from 0 to 69627904 [ 3.318183] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.323139] vdb: detected capacity change from 0 to 1073741824 [ 3.344502] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.347711] vdc: detected capacity change from 0 to 2621440000 [ 3.366912] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.371061] vdd: detected capacity change from 0 to 2621440000 [ 3.402324] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.406707] vde: detected capacity change from 0 to 4294967296 [ 3.427101] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.430224] vdf: detected capacity change from 0 to 4294967296 [ 3.440870] libphy: Fixed MDIO Bus: probed [ 3.446861] usbcore: registered new interface driver usbserial_generic [ 3.449590] usbserial: USB Serial support registered for generic [ 3.454991] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.459135] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.461551] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.463843] mousedev: PS/2 mouse device common for all mice [ 3.466961] rtc_cmos 00:05: RTC can wake from S4 [ 3.469977] rtc_cmos 00:05: registered as rtc0 [ 3.472117] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.475501] intel_pstate: CPU model not supported [ 3.479388] hid: raw HID events driver (C) Jiri Kosina [ 3.480420] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.483272] usbcore: registered new interface driver usbhid [ 3.483279] usbhid: USB HID core driver [ 3.483437] drop_monitor: Initializing network drop monitor service [ 3.483579] Initializing XFRM netlink socket [ 3.484082] NET: Registered protocol family 10 [ 3.486874] Segment Routing with IPv6 [ 3.493167] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.494367] NET: Registered protocol family 17 [ 3.497478] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.515912] mpls_gso: MPLS GSO support [ 3.522820] RAS: Correctable Errors collector initialized. [ 3.525924] AVX version of gcm_enc/dec engaged. [ 3.527773] AES CTR mode by8 optimization enabled [ 3.619997] sched_clock: Marking stable (3619971879, 0)->(4403796479, -783824600) [ 3.626680] registered taskstats version 1 [ 3.628526] Loading compiled-in X.509 certificates [ 3.632459] zswap: loaded using pool lzo/zbud [ 3.664469] Key type big_key registered [ 3.678039] Key type encrypted registered [ 3.689201] ima: No TPM chip found, activating TPM-bypass! [ 3.691745] ima: Allocated hash algorithm: sha1 [ 3.695408] ima: No architecture policies found [ 3.708260] evm: Initialising EVM extended attributes: [ 3.711011] evm: security.selinux [ 3.712484] evm: security.ima [ 3.715106] evm: security.capability [ 3.716662] evm: HMAC attrs: 0x1 [ 3.720203] rtc_cmos 00:05: setting system clock to 2026-06-04 00:33:26 UTC (1780533206) [ 3.729360] debug: unmapping init [mem 0xffffffff9ea03000-0xffffffff9ebfffff] [ 3.734674] debug: unmapping init [mem 0xffffffff9d782000-0xffffffff9da58fff] [ 3.744074] Write protecting the kernel read-only data: 28672k [ 3.749669] debug: unmapping init [mem 0xffffffff9be03000-0xffffffff9bffffff] [ 3.753336] debug: unmapping init [mem 0xffffffff9c714000-0xffffffff9c7fffff] [ 3.812130] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.819173] systemd[1]: Detected virtualization kvm. [ 3.820968] systemd[1]: Detected architecture x86-64. [ 3.824097] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.852613] systemd[1]: No hostname configured. [ 3.856719] systemd[1]: Set hostname to . [ 3.859589] random: systemd: uninitialized urandom read (16 bytes read) [ 3.864321] systemd[1]: Initializing machine ID from random generator. [ 3.918935] random: ln: uninitialized urandom read (6 bytes read) [ 4.136356] random: systemd: uninitialized urandom read (16 bytes read) [ 4.138599] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.147583] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.181660] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.256377] device-mapper: uevent: version 1.0.3 [ 5.258847] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.609244] virtio_net virtio0 ens2: renamed from eth0 [ 6.675050] random: fast init done [ 6.971251] scsi host0: ata_piix [ 7.230328] scsi host1: ata_piix [ 7.234210] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.241689] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.224303] random: crng init done [ 12.225798] random: 7 urandom warning(s) missed due to ratelimiting [ 12.690290] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 15.603579] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.336871] printk: systemd: 25 output lines suppressed due to ratelimiting [ 20.563651] SELinux: Disabled at runtime. [ 20.851835] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.877427] systemd[1]: Detected virtualization kvm. [ 20.880887] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.790434] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.802330] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.813735] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.827712] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.831570] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.857954] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.872255] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target I[ 23.235945] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS nitrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 25.266061] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ 25.417010] hrtimer: interrupt took 5677075 ns [ OK ] Started udev Kernel Device Manager. [ 26.292540] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.375353] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.823630] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.899967] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 33.311794] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit)[ 33.972098] NFS: Registering the id_resolver key type [ 33.977452] Key type id_resolver registered [ 33.980021] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg315-server login: [ 76.538612] libcfs: loading out-of-tree module taints kernel. [ 76.557957] Key type ._llcrypt registered [ 76.559585] Key type .llcrypt registered [ 76.609598] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 85.131305] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 85.828078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 85.835744] alg: No test for adler32 (adler32-zlib) [ 86.896269] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 87.248505] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 88.879223] Key type lgssc registered [ 89.567430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.581461] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 113.907370] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 119.906440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 119.919713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 121.047583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 121.065227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 121.111936] Lustre: lustre-MDT0000: new disk, initializing [ 121.158199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 121.170838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 122.955523] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 128.911219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 128.956716] Lustre: 6499:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 128.975293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 128.980177] Lustre: Skipped 1 previous similar message [ 129.041515] Lustre: lustre-MDT0001: new disk, initializing [ 129.067699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 129.081572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 129.085977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 130.655639] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 133.290345] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 137.222546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 137.334956] Lustre: lustre-OST0000: new disk, initializing [ 137.337272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 137.339857] Lustre: 8403:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 137.375562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.568191] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.856084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 143.859863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 143.881809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 145.574460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 145.636607] Lustre: lustre-OST0001: new disk, initializing [ 145.639035] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 145.642397] Lustre: 9459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 145.665808] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 147.730050] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 152.564398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 152.568838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 152.600810] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 154.234581] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 157.435689] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 159.996738] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing check_logdir /tmp/testlogs/ [ 161.837279] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing yml_node [ 163.741194] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 164.789648] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 165.934376] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 166.665756] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jun 3 20:36:10 EDT 2026 [ 174.567437] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 175.155370] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 188.384568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 188.390095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 188.391531] Lustre: Skipped 3 previous similar messages [ 188.397470] Lustre: Skipped 2 previous similar messages [ 193.458150] Lustre: server umount lustre-MDT0000 complete [ 195.085275] LustreError: 6492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533397 with bad export cookie 8303361343794021976 [ 195.086487] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 195.091202] LustreError: 6492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 198.623896] LustreError: 8145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 198.624057] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 198.624405] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 198.630364] LustreError: 8145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 198.639421] Lustre: Skipped 1 previous similar message [ 200.671880] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 200.672510] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 200.682044] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 200.684943] Lustre: Skipped 2 previous similar messages [ 204.767798] LustreError: 11891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 204.767812] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 204.774536] LustreError: 11891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 204.779993] Lustre: Skipped 1 previous similar message [ 209.375119] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 209.466449] Lustre: server umount lustre-MDT0001 complete [ 211.165173] Lustre: server umount lustre-OST0000 complete [ 212.816665] Lustre: server umount lustre-OST0001 complete [ 215.307946] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 219.265577] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 236.025502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.160118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 236.181837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 236.238134] Lustre: lustre-MDT0000: new disk, initializing [ 236.275746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.285947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 238.230627] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 244.028166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 244.082751] Lustre: 14968:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 244.242436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 246.122106] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 250.230851] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 252.047386] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 255.109769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 255.256811] Lustre: lustre-OST0000: new disk, initializing [ 255.259134] Lustre: Skipped 1 previous similar message [ 255.261824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 255.264848] Lustre: Skipped 2 previous similar messages [ 255.268585] Lustre: 16711:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 255.313844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.318177] Lustre: Skipped 1 previous similar message [ 256.328680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 256.333189] Lustre: Skipped 1 previous similar message [ 256.336720] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 256.397507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 257.932767] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 261.806503] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 263.374386] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 263.455250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 264.940556] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 265.029819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 266.721308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 266.723232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 266.724639] LustreError: Skipped 1 previous similar message [ 266.730699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 266.734642] Lustre: Skipped 1 previous similar message [ 276.964436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 276.966903] Lustre: Skipped 3 previous similar messages [ 280.031132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 280.101554] Lustre: server umount lustre-OST0000 complete [ 282.080779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.087175] Lustre: Skipped 2 previous similar messages [ 287.838940] Lustre: server umount lustre-MDT0000 complete [ 289.453492] LustreError: 15749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533492 with bad export cookie 8303361343794023509 [ 289.457294] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 289.461130] LustreError: 15749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 289.612854] Lustre: server umount lustre-MDT0001 complete [ 291.334912] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:38:14 (1780533494) [ 292.430202] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 293.170335] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:38:16 (1780533496) [ 296.042348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 296.248914] LustreError: 18975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.284151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 297.885370] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 301.537621] LustreError: 18976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.694533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 303.620872] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 306.015173] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 307.543962] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 310.543578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 313.152968] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 315.413478] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 332.258098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 332.260198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.269759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 332.272792] Lustre: Skipped 5 previous similar messages [ 335.965031] Lustre: server umount lustre-OST0000 complete [ 337.888471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 337.892240] LustreError: Skipped 1 previous similar message [ 342.496913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.504634] Lustre: Skipped 2 previous similar messages [ 352.223258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 352.279110] LustreError: 21435:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff994586cb4b00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 352.737549] LustreError: 20182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.745091] LustreError: 20182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 354.359280] Lustre: server umount lustre-MDT0000 complete [ 356.077388] LustreError: 18956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533558 with bad export cookie 8303361343794024391 [ 356.080828] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 356.082394] LustreError: 18956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 356.201798] Lustre: server umount lustre-MDT0001 complete [ 360.503305] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 361.732522] Key type lgssc unregistered [ 361.867444] LNet: 22178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.870910] LNetError: 22178:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 361.883386] LNet: Removed LNI 192.168.203.115@tcp [ 362.276145] Key type .llcrypt unregistered [ 362.277889] Key type ._llcrypt unregistered [ 372.500443] Key type ._llcrypt registered [ 372.502179] Key type .llcrypt registered [ 372.556697] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 372.888197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 372.895938] alg: No test for adler32 (adler32-zlib) [ 373.777920] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 373.884876] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 375.479191] Key type lgssc registered [ 375.979199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 379.751116] loop: module loaded [ 379.782179] loop0: detected capacity change from 0 to 204800000 [ 379.882715] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 379.888692] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 380.026855] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 382.673970] loop0: detected capacity change from 0 to 204800000 [ 382.749812] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 382.755721] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 382.761924] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 385.024143] loop0: detected capacity change from 0 to 204800000 [ 387.466876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 387.480229] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.602679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 388.625821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 388.667793] Lustre: lustre-MDT0000: new disk, initializing [ 388.709727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.718499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 390.332273] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 394.857707] loop1: detected capacity change from 0 to 204800000 [ 397.273925] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 397.436082] Lustre: lustre-OST07c6: new disk, initializing [ 397.437841] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 397.441101] Lustre: Skipped 1 previous similar message [ 397.444166] Lustre: 25311:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 397.481107] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 399.971109] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 402.952648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 402.957256] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 402.976826] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 407.649667] Lustre: 24314:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 407.654924] Lustre: 24314:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 407.660179] Lustre: 24314:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 407.664391] Lustre: 24314:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 407.668338] Lustre: 24314:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 407.672047] Lustre: 24314:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 420.631989] Lustre: 24312:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 420.637176] Lustre: 24312:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 420.640399] Lustre: 24312:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 420.643802] Lustre: 24312:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 420.647587] Lustre: 24312:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 420.650889] Lustre: 24312:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 424.402981] Lustre: 24893:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 424.409412] Lustre: 24893:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 424.413166] Lustre: 24893:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 424.417896] Lustre: 24893:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 424.421839] Lustre: 24893:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 424.425132] Lustre: 24893:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 424.428168] Lustre: 24893:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 424.432150] Lustre: 24893:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 424.436039] Lustre: 24893:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 424.439165] Lustre: 24893:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 424.442173] Lustre: 24893:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 424.445161] Lustre: 24893:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 427.531643] Lustre: 24312:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 427.535951] Lustre: 24312:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 427.541620] Lustre: 24312:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 427.544946] Lustre: 24312:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 427.549372] Lustre: 24312:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 427.554339] Lustre: 24312:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 427.559768] Lustre: 24312:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 427.565502] Lustre: 24312:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 427.570103] Lustre: 24312:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 427.574192] Lustre: 24312:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 427.577675] Lustre: 24312:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 427.581012] Lustre: 24312:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 434.288703] Lustre: 24893:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 434.295692] Lustre: 24893:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 434.300532] Lustre: 24893:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 434.305106] Lustre: 24893:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 434.310844] Lustre: 24893:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 434.314651] Lustre: 24893:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 434.319294] Lustre: 24893:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 434.322370] Lustre: 24893:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 434.325393] Lustre: 24893:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 434.329660] Lustre: 24893:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 434.333492] Lustre: 24893:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 434.338455] Lustre: 24893:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 444.186470] Lustre: 24893:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 444.189650] Lustre: 24893:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 444.192925] Lustre: 24893:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 444.196254] Lustre: 24893:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 444.199462] Lustre: 24893:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 444.203529] Lustre: 24893:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 444.207584] Lustre: 24893:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 444.212362] Lustre: 24893:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 444.218504] Lustre: 24893:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 444.222314] Lustre: 24893:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 444.227041] Lustre: 24893:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 444.230441] Lustre: 24893:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 489.953778] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 489.957485] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.964598] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 495.072682] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 500.191326] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 500.192332] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 500.262452] Lustre: server umount lustre-OST07c6 complete [ 504.642202] Lustre: server umount lustre-MDT0000 complete [ 511.787787] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 513.032504] Key type lgssc unregistered [ 513.186599] LNet: 27931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.190351] LNetError: 27931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 513.200782] LNet: Removed LNI 192.168.203.115@tcp [ 513.557819] Key type .llcrypt unregistered [ 513.559213] Key type ._llcrypt unregistered [ 522.605557] Key type ._llcrypt registered [ 522.606795] Key type .llcrypt registered [ 522.655706] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 523.082687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 523.115327] alg: No test for adler32 (adler32-zlib) [ 524.007416] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 524.127863] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 525.727176] Key type lgssc registered [ 526.165633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 529.699470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 529.712971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.942509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.575417] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 536.397092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.575510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 538.259500] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 540.610226] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 542.231991] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 545.181723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 545.349158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.599312] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 549.797269] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 555.627279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 564.198331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 564.202350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.210135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 564.391667] Lustre: server umount lustre-OST0000 complete [ 566.240817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.242042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 566.247665] Lustre: Skipped 1 previous similar message [ 571.363262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 571.366249] Lustre: Skipped 2 previous similar messages [ 572.411105] Lustre: server umount lustre-MDT0000 complete [ 574.229443] LustreError: 29145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533777 with bad export cookie 15099164701523219058 [ 574.233773] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.237369] LustreError: 29145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 584.764224] Lustre: server umount lustre-MDT0001 complete [ 589.732554] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 591.128484] Key type lgssc unregistered [ 591.281662] LNet: 32542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.285961] LNetError: 32542:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 591.301886] LNet: Removed LNI 192.168.203.115@tcp [ 591.675115] Key type .llcrypt unregistered [ 591.678336] Key type ._llcrypt unregistered [ 600.616234] Key type ._llcrypt registered [ 600.617759] Key type .llcrypt registered [ 600.671946] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 601.064803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 601.073653] alg: No test for adler32 (adler32-zlib) [ 601.992797] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 602.111876] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 603.711158] Key type lgssc registered [ 604.219399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 607.676393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 607.686479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.914358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 610.604851] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 614.269889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.456472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 616.245808] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 618.558273] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 620.227830] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 623.171694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 623.344667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.925990] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 628.321736] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 633.577325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 639.970298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 639.974099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 639.985263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 644.067601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.067840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 644.068653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 645.284841] Lustre: server umount lustre-OST0000 complete [ 649.186065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.196230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 649.199103] Lustre: Skipped 2 previous similar messages [ 653.032864] Lustre: server umount lustre-MDT0000 complete [ 654.305545] LustreError: 34495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 654.313040] LustreError: 34495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 654.734059] LustreError: 33753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533857 with bad export cookie 17197149492906973728 [ 654.737086] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 654.741780] LustreError: 33753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 654.905210] Lustre: server umount lustre-MDT0001 complete [ 659.321995] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 660.620971] Key type lgssc unregistered [ 660.794553] LNet: 36950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.799507] LNetError: 36950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.809359] LNet: Removed LNI 192.168.203.115@tcp [ 661.195153] Key type .llcrypt unregistered [ 661.196322] Key type ._llcrypt unregistered [ 672.027887] Key type ._llcrypt registered [ 672.029767] Key type .llcrypt registered [ 672.079954] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 672.579232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.597522] alg: No test for adler32 (adler32-zlib) [ 673.490259] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 673.598989] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 675.207171] Key type lgssc registered [ 675.723932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 678.980581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 678.993228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.221653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.926829] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 685.706629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 685.900084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 687.552508] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 689.928276] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 691.489672] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 694.429620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 694.615094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 697.165907] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 699.665756] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 704.871871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 710.115538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 710.119090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 710.121163] Lustre: Skipped 1 previous similar message [ 713.575622] Lustre: server umount lustre-MDT0000 complete [ 715.233407] LustreError: 38324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 715.242485] LustreError: 38324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 715.254421] LustreError: 38311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533918 with bad export cookie 15066245587923340918 [ 715.257615] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 715.260821] LustreError: 38311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 720.352331] LustreError: 38326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.353458] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 720.370598] Lustre: Skipped 1 previous similar message [ 720.374704] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 720.380377] Lustre: Skipped 3 previous similar messages [ 721.491983] Lustre: server umount lustre-MDT0001 complete [ 740.018713] Lustre: server umount lustre-OST0000 complete [ 747.235895] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 748.521844] Key type lgssc unregistered [ 748.682761] LNet: 41835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.686892] LNetError: 41835:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.700539] LNet: Removed LNI 192.168.203.115@tcp [ 749.052877] Key type .llcrypt unregistered [ 749.054629] Key type ._llcrypt unregistered [ 759.898802] Key type ._llcrypt registered [ 759.900625] Key type .llcrypt registered [ 759.949369] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 760.338421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 760.373360] alg: No test for adler32 (adler32-zlib) [ 761.268231] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 761.384371] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 762.976537] Key type lgssc registered [ 763.483344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 766.864816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 766.873061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.084215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 769.723306] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 773.287116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 773.461921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 775.180914] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 777.540939] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 779.020708] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 781.742647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 781.894670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 784.232151] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 786.462784] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 793.190791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 803.308620] Lustre: lustre-OST0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 803.810327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 803.814146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 803.814411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 803.816800] Lustre: Skipped 1 previous similar message [ 806.696961] Lustre: server umount lustre-OST0000 complete [ 820.008358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.012778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 820.015284] Lustre: Skipped 2 previous similar messages [ 824.098515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 824.100795] Lustre: Skipped 1 previous similar message [ 826.232392] Lustre: server umount lustre-MDT0000 complete [ 827.877402] LustreError: 43930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534030 with bad export cookie 9773322065610040507 [ 827.878462] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 827.882580] LustreError: 43930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 842.207118] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 842.347260] Lustre: server umount lustre-MDT0001 complete [ 846.888262] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 848.284206] Key type lgssc unregistered [ 848.438480] LNet: 46585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.446152] LNetError: 46585:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.509970] LNet: Removed LNI 192.168.203.115@tcp [ 849.902364] Key type .llcrypt unregistered [ 849.903979] Key type ._llcrypt unregistered [ 860.781817] Key type ._llcrypt registered [ 860.783530] Key type .llcrypt registered [ 860.824066] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 861.192023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 861.200096] alg: No test for adler32 (adler32-zlib) [ 862.073707] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 862.174887] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 863.775310] Key type lgssc registered [ 864.372243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 867.764215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 867.773382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.040508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 870.769436] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 874.400864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 874.587026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 876.229965] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 878.502450] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 879.995658] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 882.843742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 883.001240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 885.584584] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 888.060976] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 894.247447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 900.902906] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 902.343643] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 913.208079] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-0320h48m36s [ 913.886815] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 914.912864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 914.913855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 914.919108] Lustre: Skipped 2 previous similar messages [ 914.921627] Lustre: Skipped 1 previous similar message [ 915.939030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 915.942140] Lustre: Skipped 3 previous similar messages [ 919.016194] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 922.081514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 922.084429] Lustre: Skipped 3 previous similar messages [ 928.227381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 928.230368] Lustre: Skipped 3 previous similar messages [ 929.247952] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 929.317172] Lustre: server umount lustre-MDT0000 complete [ 930.695835] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 931.295088] LustreError: 47960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 931.304014] LustreError: 47960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 933.247745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 933.311611] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 933.418449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 933.445536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 934.438488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 934.849978] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 936.094650] Lustre: DEBUG MARKER: Wait for df (35981) ... [ 936.696471] Lustre: DEBUG MARKER: done [ 938.464349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 938.468349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 938.476934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 939.490202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 939.862093] Lustre: server umount lustre-OST0000 complete [ 944.607886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 944.611528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 944.617393] Lustre: Skipped 1 previous similar message [ 947.427393] Lustre: server umount lustre-MDT0000 complete [ 948.889454] LustreError: 47945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534151 with bad export cookie 3074073745689379270 [ 948.893835] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 948.893957] LustreError: 47945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 949.021382] Lustre: server umount lustre-MDT0001 complete [ 952.965542] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 954.080470] Key type lgssc unregistered [ 954.213473] LNet: 52530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.218134] LNetError: 52530:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.227538] LNet: Removed LNI 192.168.203.115@tcp [ 954.542184] Key type .llcrypt unregistered [ 954.543713] Key type ._llcrypt unregistered [ 969.874825] Key type ._llcrypt registered [ 969.875969] Key type .llcrypt registered [ 969.913900] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 970.318561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 970.357966] alg: No test for adler32 (adler32-zlib) [ 971.222154] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 971.315141] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 972.903220] Key type lgssc registered [ 973.339762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 976.263245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 976.273407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.404664] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 977.417312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 977.489050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.959018] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 982.061492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 982.125076] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 982.140378] Lustre: 54701:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 982.251683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 983.850741] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 986.128040] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 987.651659] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 990.410600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 990.503484] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 990.600853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.608588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 992.892356] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 995.121060] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 998.712213] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-0320h50m02s [ 999.390736] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1002.380126] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1003.395677] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1003.415602] LustreError: 54707:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1023.469822] Lustre: lustre-MDT0000: Client e52941dc-c163-4d9c-ad81-9f44876eea96 (at 192.168.203.15@tcp) reconnecting [ 1043.463156] LustreError: 54707:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1044.241530] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1045.125012] Lustre: DEBUG MARKER: done [ 1049.057101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1049.059146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.060519] LustreError: Skipped 1 previous similar message [ 1049.065765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1049.069534] Lustre: Skipped 1 previous similar message [ 1052.819206] Lustre: server umount lustre-OST0000 complete [ 1059.297021] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.297243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1059.297799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1059.302245] Lustre: Skipped 2 previous similar messages [ 1060.320883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1060.324853] Lustre: Skipped 2 previous similar messages [ 1060.389465] Lustre: server umount lustre-MDT0000 complete [ 1061.709545] LustreError: 54691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534264 with bad export cookie 3173814666242174751 [ 1061.714546] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1061.715085] LustreError: 54691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1061.836470] Lustre: server umount lustre-MDT0001 complete [ 1065.692846] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1066.805909] Key type lgssc unregistered [ 1066.948445] LNet: 58669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.951435] LNetError: 58669:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.963343] LNet: Removed LNI 192.168.203.115@tcp [ 1067.268735] Key type .llcrypt unregistered [ 1067.269949] Key type ._llcrypt unregistered [ 1083.940598] Key type ._llcrypt registered [ 1083.941529] Key type .llcrypt registered [ 1083.977818] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1084.398873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1084.420642] alg: No test for adler32 (adler32-zlib) [ 1085.268127] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1085.351783] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1086.935118] Key type lgssc registered [ 1087.315810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1088.515750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1088.523194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.627080] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1089.639162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1089.711713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.073822] Lustre: Failing over lustre-MDT0000 [ 1090.193830] Lustre: server umount lustre-MDT0000 complete [ 1092.546846] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:51:36 (1780534296) [ 1094.945022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1095.120664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1096.553559] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1099.719083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.757950] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1099.771511] Lustre: 61428:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1099.882666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1101.374216] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1103.402250] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1104.819692] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1107.393154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1107.478575] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1107.558640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.381937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1109.633083] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1111.648213] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1115.931327] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1118.691107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.693069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1118.699696] Lustre: Skipped 2 previous similar messages [ 1118.704338] Lustre: Skipped 2 previous similar messages [ 1123.808652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.812498] Lustre: Skipped 2 previous similar messages [ 1124.140362] Lustre: server umount lustre-MDT0000 complete [ 1125.822417] LustreError: 61418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534328 with bad export cookie 14784059423970403667 [ 1125.825704] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1125.829075] LustreError: 61418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1125.971657] Lustre: server umount lustre-MDT0001 complete [ 1127.434681] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1129.229955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1132.364528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1136.096120] LustreError: 60350:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007f1470de x1867025521606272/t0(0) o250->MGC192.168.203.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1136.267583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1136.304081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1137.737519] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1140.841566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.936524] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.500150] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1144.381910] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1144.415066] Lustre: 60354:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780534331/real 1780534331] req@000000009e7874ec x1867025521605888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780534347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1145.666192] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1146.341058] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1147.359408] Lustre: 60354:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780534333/real 1780534333] req@000000004ec778aa x1867025521606016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780534349 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1148.387843] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1149.407100] Lustre: 60351:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780534336/real 1780534336] req@00000000b76a071e x1867025521606144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780534352 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1151.455290] Lustre: 60351:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780534338/real 1780534338] req@00000000c8a08481 x1867025521606400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780534354 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1153.507311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.508734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.513395] Lustre: Skipped 1 previous similar message [ 1153.517771] Lustre: Skipped 2 previous similar messages [ 1158.624548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1158.627036] Lustre: Skipped 2 previous similar messages [ 1158.949909] Lustre: server umount lustre-MDT0000 complete [ 1160.447156] LustreError: 62146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534363 with bad export cookie 14784059423970405935 [ 1160.451755] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1160.453656] LustreError: 62146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1160.568385] Lustre: server umount lustre-MDT0001 complete [ 1162.926452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1166.402521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.911712] LustreError: 60350:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f29c8fad x1867025521626880/t0(0) o250->MGC192.168.203.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1171.035814] LustreError: 63051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.036592] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1171.044262] LustreError: 63051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1171.050119] Lustre: Skipped 1 previous similar message [ 1171.071098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1171.073867] Lustre: Skipped 1 previous similar message [ 1171.096257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1172.556222] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1175.907502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.628070] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1179.615090] Lustre: 60353:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780534366/real 1780534366] req@00000000aec28c2c x1867025521626496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780534382 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1179.669895] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1181.051150] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1181.159850] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1183.968994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.971142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.975639] Lustre: Skipped 1 previous similar message [ 1189.086541] Lustre: server umount lustre-MDT0000 complete [ 1189.344578] LustreError: 68569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.349758] LustreError: 68569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1190.364258] LustreError: 61420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534393 with bad export cookie 14784059423970407314 [ 1190.366991] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1190.368419] LustreError: 61420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1190.463374] Lustre: server umount lustre-MDT0001 complete [ 1192.662851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1193.880925] Lustre: DEBUG MARKER: files compared the same [ 1204.894409] Lustre: server umount lustre-OST0000 complete [ 1210.515269] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1211.573331] Key type lgssc unregistered [ 1211.713515] LNet: 70340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1211.716662] LNetError: 70340:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1211.726412] LNet: Removed LNI 192.168.203.115@tcp [ 1212.040125] Key type .llcrypt unregistered [ 1212.041892] Key type ._llcrypt unregistered [ 1221.244205] Key type ._llcrypt registered [ 1221.245537] Key type .llcrypt registered [ 1221.285708] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1221.715360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1221.742449] alg: No test for adler32 (adler32-zlib) [ 1222.609527] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1222.694314] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1224.271132] Key type lgssc registered [ 1224.631592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1227.194302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1227.201500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.360538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.572139] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug +malloc all [ 1232.344652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.441382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1233.695423] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug +malloc all [ 1235.503604] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1236.641627] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1238.932027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1239.047695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.967034] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug +malloc all [ 1242.750311] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1253.408992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1263.584038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.584074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1263.587547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1263.589867] LustreError: Skipped 1 previous similar message [ 1267.013875] Lustre: server umount lustre-OST0000 complete [ 1268.703528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.707815] Lustre: Skipped 1 previous similar message [ 1268.709162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1268.710872] Lustre: Skipped 1 previous similar message [ 1273.311712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1273.315503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.322033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1274.382798] Lustre: server umount lustre-MDT0000 complete [ 1275.546651] LustreError: 71699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534478 with bad export cookie 8063363273711903300 [ 1275.548067] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1275.550231] LustreError: 71699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1275.653049] Lustre: server umount lustre-MDT0001 complete [ 1279.216236] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1280.175353] Key type lgssc unregistered [ 1280.292316] LNet: 74916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.294235] LNetError: 74916:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.301293] LNet: Removed LNI 192.168.203.115@tcp [ 1280.561891] Key type .llcrypt unregistered [ 1280.563431] Key type ._llcrypt unregistered [ 1289.000176] Key type ._llcrypt registered [ 1289.001769] Key type .llcrypt registered [ 1289.040238] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1289.360541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1289.404043] alg: No test for adler32 (adler32-zlib) [ 1290.263713] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1290.349182] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1291.927159] Key type lgssc registered [ 1292.270401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1294.793806] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1294.801215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1310.239265] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1315.359244] LustreError: 76267:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 1341.983385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1343.749851] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1345.463553] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1348.007856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.043522] Lustre: *** cfs_fail_loc=706, val=0*** [ 1348.176124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1348.201843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1349.396290] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1352.080277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.198655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1353.505271] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1355.175898] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1356.304378] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1357.280699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1357.283947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.290290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1358.303755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1358.305975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.311746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1359.050310] Lustre: server umount lustre-OST0000 complete [ 1361.376666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.381726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1361.383535] Lustre: Skipped 1 previous similar message [ 1366.355827] Lustre: server umount lustre-MDT0000 complete [ 1366.495918] LustreError: 78787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1366.499984] LustreError: 78787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1367.522135] LustreError: 76274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534570 with bad export cookie 6215483766235555709 [ 1367.524591] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1367.527543] LustreError: 76274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1367.636713] Lustre: server umount lustre-MDT0001 complete [ 1370.973550] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1371.917850] Key type lgssc unregistered [ 1372.027351] LNet: 79523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.029513] LNetError: 79523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.037258] LNet: Removed LNI 192.168.203.115@tcp [ 1372.295153] Key type .llcrypt unregistered [ 1372.296105] Key type ._llcrypt unregistered [ 1380.680713] Key type ._llcrypt registered [ 1380.682286] Key type .llcrypt registered [ 1380.716545] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1381.121752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1381.130589] alg: No test for adler32 (adler32-zlib) [ 1381.981278] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1382.053297] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1383.631101] Key type lgssc registered [ 1383.945429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1386.267200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1386.275184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.489773] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1390.812787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.988835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1392.360626] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1395.121852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1395.244169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.983288] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1399.485934] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1399.490517] Lustre: MGS: Client 7d551af2-9149-4aa4-86f5-1d7327d7b2cf (at 0@lo) reconnecting [ 1399.498283] LustreError: 81583:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1399.578624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1399.611092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1400.847230] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1410.016304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1410.019031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1410.022929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1411.042599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1411.042751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1411.049636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1414.672799] Lustre: server umount lustre-OST0000 complete [ 1416.161038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1416.161561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.162157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1416.162163] Lustre: Skipped 1 previous similar message [ 1416.177625] Lustre: Skipped 1 previous similar message [ 1421.280294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1421.282256] Lustre: Skipped 2 previous similar messages [ 1422.083800] Lustre: server umount lustre-MDT0000 complete [ 1423.534692] Lustre: server umount lustre-MDT0001 complete [ 1424.840527] Lustre: server umount MGS complete [ 1428.566467] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:57:12 (1780534632) [ 1437.167100] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 1439.853448] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1450.644360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.989852] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1455.875630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1455.968203] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1455.988288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1456.031974] Lustre: lustre-MDT0001: new disk, initializing [ 1456.061486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1456.070811] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1456.073444] Lustre: 88630:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1456.076272] LustreError: 88630:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1457.319984] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1461.199512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1461.289619] Lustre: lustre-OST0000: new disk, initializing [ 1461.291810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1461.294318] Lustre: 89381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1461.311960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.084826] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1465.977956] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1468.033507] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1468.037682] Lustre: MGS: Client 4cac04b0-8088-45fd-a9d1-df5af471ae64 (at 0@lo) reconnecting [ 1468.051806] Lustre: 90308:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1468.058172] LustreError: 90308:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1468.066466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1468.097113] Lustre: lustre-MDT0000: new disk, initializing [ 1468.141967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1468.343484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1469.153562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1469.156201] Lustre: Skipped 1 previous similar message [ 1469.157789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1469.218663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1469.326206] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1488.867589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1488.867778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.872649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1488.875355] Lustre: Skipped 1 previous similar message [ 1488.878873] Lustre: Skipped 1 previous similar message [ 1499.105834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1499.107474] Lustre: Skipped 3 previous similar messages [ 1502.175116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1502.231291] Lustre: server umount lustre-OST0000 complete [ 1504.225156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1504.225232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1504.229110] Lustre: Skipped 1 previous similar message [ 1504.232810] LustreError: Skipped 1 previous similar message [ 1509.763050] Lustre: server umount lustre-MDT0000 complete [ 1517.232626] Lustre: server umount lustre-MDT0001 complete [ 1518.509387] Lustre: server umount MGS complete [ 1521.990275] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:58:45 (1780534725) [ 1527.716846] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1528.752381] Key type lgssc unregistered [ 1528.880336] LNet: 92800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.882903] LNetError: 92800:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.891333] LNet: Removed LNI 192.168.203.115@tcp [ 1529.224232] Key type .llcrypt unregistered [ 1529.226135] Key type ._llcrypt unregistered [ 1536.227185] Key type ._llcrypt registered [ 1536.228233] Key type .llcrypt registered [ 1536.268514] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1536.700439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1536.757201] alg: No test for adler32 (adler32-zlib) [ 1537.628106] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1537.715661] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1539.295138] Key type lgssc registered [ 1539.681456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1542.212210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1542.218176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.350127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.541976] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1547.480975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1547.592857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1548.801466] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1550.504833] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1551.680714] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1553.838247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1553.889423] LustreError: 95576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.894229] LustreError: 95576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1553.939434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1555.697370] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1557.373021] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1558.059432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1568.226330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1568.227125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.228394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1568.235228] Lustre: Skipped 1 previous similar message [ 1571.979657] Lustre: server umount lustre-OST0000 complete [ 1573.344605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1573.344631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1573.345200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1573.345205] Lustre: Skipped 1 previous similar message [ 1573.348856] Lustre: Skipped 1 previous similar message [ 1573.350786] LustreError: Skipped 1 previous similar message [ 1578.463977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1578.465550] Lustre: Skipped 2 previous similar messages [ 1579.279308] Lustre: server umount lustre-MDT0000 complete [ 1580.475211] LustreError: 93962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534783 with bad export cookie 12822082240188156923 [ 1580.477313] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1580.478012] LustreError: 93962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1580.583366] Lustre: server umount lustre-MDT0001 complete [ 1583.693863] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1584.663615] Key type lgssc unregistered [ 1584.791552] LNet: 97157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.793420] LNetError: 97157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.800191] LNet: Removed LNI 192.168.203.115@tcp [ 1585.067955] Key type .llcrypt unregistered [ 1585.069052] Key type ._llcrypt unregistered [ 1590.838673] Key type ._llcrypt registered [ 1590.839485] Key type .llcrypt registered [ 1590.873352] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1591.145702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1591.186859] alg: No test for adler32 (adler32-zlib) [ 1592.030180] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1592.115290] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1593.695121] Key type lgssc registered [ 1594.003249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1595.938195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1595.938321] LustreError: 98202:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1595.940518] Lustre: Skipped 1 previous similar message [ 1595.944779] LustreError: 98201:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1595.946736] LustreError: 98202:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1595.948796] LustreError: 98202:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1595.951752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1595.952300] LustreError: 98202:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1597.091353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.373284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1599.458084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1600.602040] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1602.026391] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing lsmod [ 1602.882465] LustreError: 99371:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1602.890115] LustreError: 99374:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1602.892676] LustreError: 99371:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 1602.894811] LustreError: 99371:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1602.897506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1602.898192] LustreError: 99371:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1603.004823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.906796] LustreError: 98268:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1603.911944] LustreError: 98268:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1603.916684] LustreError: 98268:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1604.575729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1604.578626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.583593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1608.161442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.161487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1608.165166] Lustre: Skipped 1 previous similar message [ 1608.168156] Lustre: Skipped 2 previous similar messages [ 1610.073984] Lustre: server umount lustre-MDT0000 complete [ 1610.210215] LustreError: 98672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1610.215570] LustreError: 98672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1611.233958] LustreError: 98213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534814 with bad export cookie 12844054602961186887 [ 1611.237070] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1611.241262] LustreError: 98213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1612.255214] LustreError: 98669:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000040a5d4b4 x1867026052168960/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1612.305428] Lustre: server umount lustre-MDT0001 complete [ 1623.578643] Lustre: server umount lustre-OST0000 complete [ 1626.276337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.422177] LustreError: 100439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1626.442562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.681100] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1630.348477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.467151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1631.694556] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1633.351069] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1634.528394] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1636.607908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1637.735623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1638.454902] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1640.104976] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1642.975785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1642.978713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1642.982619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1646.050502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1646.051230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1646.052976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.405688] Lustre: server umount lustre-OST0000 complete [ 1651.170455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1651.170556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.171401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1651.171406] Lustre: Skipped 1 previous similar message [ 1651.179678] Lustre: Skipped 1 previous similar message [ 1655.762421] Lustre: server umount lustre-MDT0000 complete [ 1656.287916] LustreError: 101153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.293167] LustreError: 101153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1656.959719] LustreError: 100419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534859 with bad export cookie 12844054602961187566 [ 1656.963377] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1656.965601] LustreError: 100419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1657.081655] Lustre: server umount lustre-MDT0001 complete [ 1660.417272] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1661.423940] Key type lgssc unregistered [ 1661.562352] LNet: 103630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.565802] LNetError: 103630:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.574447] LNet: Removed LNI 192.168.203.115@tcp [ 1661.856983] Key type .llcrypt unregistered [ 1661.857903] Key type ._llcrypt unregistered [ 1670.350049] Key type ._llcrypt registered [ 1670.351642] Key type .llcrypt registered [ 1670.382981] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1670.650242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1670.695058] alg: No test for adler32 (adler32-zlib) [ 1671.537734] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1671.620569] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1673.199162] Key type lgssc registered [ 1673.516677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1675.841258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1675.847442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.003692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1678.265227] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1681.064229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.193698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1682.520196] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1684.255393] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1685.408734] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1687.611053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1687.736121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.643628] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1691.444176] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1702.055504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1708.948767] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1709.717840] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1711.584127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1711.587036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.591540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1712.608764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1712.609678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1712.609961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1716.747218] Lustre: server umount lustre-OST0000 complete [ 1722.336247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1722.339251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.343430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1722.346188] Lustre: Skipped 1 previous similar message [ 1724.063284] Lustre: server umount lustre-MDT0000 complete [ 1725.356756] LustreError: 104992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534928 with bad export cookie 5302643045761489298 [ 1725.357815] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1725.359792] LustreError: 104992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1725.469959] Lustre: server umount lustre-MDT0001 complete [ 1728.846456] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1729.871348] Key type lgssc unregistered [ 1729.998350] LNet: 108314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.000618] LNetError: 108314:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.009302] LNet: Removed LNI 192.168.203.115@tcp [ 1730.311974] Key type .llcrypt unregistered [ 1730.313124] Key type ._llcrypt unregistered [ 1736.931697] Key type ._llcrypt registered [ 1736.933225] Key type .llcrypt registered [ 1736.963524] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1737.304844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1737.323975] alg: No test for adler32 (adler32-zlib) [ 1738.163943] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1738.237407] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1739.807178] Key type lgssc registered [ 1740.202771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1742.687489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1742.694518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.836190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.992510] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1747.661604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1747.784125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1748.976015] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1750.646096] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1751.723343] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1753.769436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1753.872174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.919791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1755.638112] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1757.343959] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1763.299545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1763.300526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1763.304440] Lustre: Skipped 1 previous similar message [ 1763.307833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1766.926446] Lustre: server umount lustre-OST0000 complete [ 1768.416792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1768.417278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1768.425630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1768.429398] Lustre: Skipped 2 previous similar messages [ 1773.536230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1773.538740] Lustre: Skipped 1 previous similar message [ 1774.231828] Lustre: server umount lustre-MDT0000 complete [ 1775.459079] LustreError: 110216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534978 with bad export cookie 5237082575943693511 [ 1775.461241] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1775.463667] LustreError: 110216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1775.577338] Lustre: server umount lustre-MDT0001 complete [ 1779.059096] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1779.953601] Key type lgssc unregistered [ 1780.075309] LNet: 112677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.078506] LNetError: 112677:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.085243] LNet: Removed LNI 192.168.203.115@tcp [ 1780.338480] Key type .llcrypt unregistered [ 1780.339949] Key type ._llcrypt unregistered [ 1788.796164] Key type ._llcrypt registered [ 1788.797859] Key type .llcrypt registered [ 1788.841359] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1789.190951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1789.198114] alg: No test for adler32 (adler32-zlib) [ 1790.062475] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1790.157758] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1791.743184] Key type lgssc registered [ 1792.190836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1794.847830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1794.854522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.013230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.320500] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1800.222624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.348398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1801.686695] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1803.529064] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1804.836323] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1807.228367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1807.346239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1809.248248] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1810.826052] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1820.646807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1823.649951] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1824.858606] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1824.860884] Lustre: Skipped 2 previous similar messages [ 1826.221320] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1826.225468] Lustre: Skipped 2 previous similar messages [ 1832.958420] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1832.969793] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1834.972169] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.15@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1834.975939] Lustre: Skipped 2 previous similar messages [ 1842.679930] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.15@tcp 192.168.0.[2,10]@tcp [ 1844.092987] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.15/24@tcp in log lustre-MDT0000 [ 1844.095497] Lustre: Skipped 2 previous similar messages [ 1852.411038] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.15/24@tcp [ 1852.413192] Lustre: Skipped 1 previous similar message [ 1856.482385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1856.482814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.484717] LustreError: Skipped 1 previous similar message [ 1856.489578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1856.492070] Lustre: Skipped 1 previous similar message [ 1860.760933] Lustre: server umount lustre-OST0000 complete [ 1862.111748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1862.113617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.117138] Lustre: Skipped 1 previous similar message [ 1862.118970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.720148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.720908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1868.115593] Lustre: server umount lustre-MDT0000 complete [ 1869.269123] LustreError: 114036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535072 with bad export cookie 5170119534268366942 [ 1869.272218] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1869.273600] LustreError: 114036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1869.383783] Lustre: server umount lustre-MDT0001 complete [ 1872.776124] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1873.698108] Key type lgssc unregistered [ 1873.820366] LNet: 119131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.822766] LNetError: 119131:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.829234] LNet: Removed LNI 192.168.203.115@tcp [ 1874.090440] Key type .llcrypt unregistered [ 1874.091534] Key type ._llcrypt unregistered [ 1883.700032] Key type ._llcrypt registered [ 1883.701102] Key type .llcrypt registered [ 1883.732473] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1884.146599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1884.159072] alg: No test for adler32 (adler32-zlib) [ 1885.005715] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1885.077860] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1886.655163] Key type lgssc registered [ 1886.986633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1889.375617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1889.382929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.510949] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1890.514565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.517675] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.15/24@tcp [ 1891.700383] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1894.325479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.428100] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1894.435704] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.15/24@tcp [ 1894.451377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1895.627536] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1897.244356] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1898.324852] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1900.365757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1900.466097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.478820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 1902.096115] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1903.666596] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1909.729177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1909.730991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.732424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1909.736180] Lustre: Skipped 1 previous similar message [ 1913.618459] Lustre: server umount lustre-OST0000 complete [ 1914.852059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1914.854370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.855330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1914.860971] Lustre: Skipped 1 previous similar message [ 1914.865009] Lustre: Skipped 2 previous similar messages [ 1919.968100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1919.970202] Lustre: Skipped 1 previous similar message [ 1925.088235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1925.090802] Lustre: Skipped 1 previous similar message [ 1929.183253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1929.260427] Lustre: server umount lustre-MDT0000 complete [ 1930.208296] LustreError: 120672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1930.214755] LustreError: 120672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1930.559857] LustreError: 120634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535133 with bad export cookie 3675635184394691540 [ 1930.562583] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1930.566525] LustreError: 120634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1930.675570] Lustre: server umount lustre-MDT0001 complete [ 1934.246420] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1935.318793] Key type lgssc unregistered [ 1935.437393] LNet: 123884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1935.440492] LNetError: 123884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1935.449284] LNet: Removed LNI 192.168.203.115@tcp [ 1935.731244] Key type .llcrypt unregistered [ 1935.732221] Key type ._llcrypt unregistered [ 1943.192244] Key type ._llcrypt registered [ 1943.193165] Key type .llcrypt registered [ 1943.223979] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 1948.715851] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1949.068382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1949.073986] alg: No test for adler32 (adler32-zlib) [ 1949.913511] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1949.980937] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1951.551107] Key type lgssc registered [ 1951.909081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1962.600490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1962.605171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.697685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1963.709155] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1963.745315] Lustre: lustre-MDT0000: new disk, initializing [ 1963.767550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.774851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1964.983402] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1969.018349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.059285] Lustre: 127224:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1969.081523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1969.083267] Lustre: Skipped 1 previous similar message [ 1969.117883] Lustre: lustre-MDT0001: new disk, initializing [ 1969.150326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.161881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1969.165121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1970.400655] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1973.363273] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1974.539360] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1976.710370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1976.803767] Lustre: lustre-OST0000: new disk, initializing [ 1976.805601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1976.809613] Lustre: 128965:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1976.840286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.683494] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1981.687773] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1981.946996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1981.950172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1981.983046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1983.052919] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1983.118110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1984.425571] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1984.500475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1987.041411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1987.043134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.043921] LustreError: Skipped 1 previous similar message [ 1987.048234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1987.050831] Lustre: Skipped 1 previous similar message [ 1991.249911] Lustre: server umount lustre-OST0000 complete [ 1994.719908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1994.723457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.728923] Lustre: Skipped 1 previous similar message [ 1994.731345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.280085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.280619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.675755] Lustre: server umount lustre-MDT0000 complete [ 1999.846106] LustreError: 128001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535202 with bad export cookie 1458746876418611637 [ 1999.848973] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1999.849460] LustreError: 128001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1999.948640] Lustre: server umount lustre-MDT0001 complete [ 2002.648531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.804748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.995311] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2006.571281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.871708] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2009.459072] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2010.513764] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2012.575962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2012.677239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2012.680076] Lustre: Skipped 1 previous similar message [ 2014.406878] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2016.010966] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2017.264433] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2017.328383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2018.513413] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2018.585581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2022.008093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2022.077125] Lustre: lustre-OST0001: new disk, initializing [ 2022.079489] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2022.082755] Lustre: 134030:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2022.118445] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2023.616451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2023.620474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2023.651817] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2023.975674] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2026.965641] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2027.028289] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2028.193396] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2028.255907] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2029.276348] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90bec7a6e800.ost_server_uuid 50 [ 2029.738491] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90bec7a6e800.ost_server_uuid in FULL state after 0 sec [ 2034.147589] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2034.147858] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.148829] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2034.148833] Lustre: Skipped 1 previous similar message [ 2034.149702] LustreError: Skipped 1 previous similar message [ 2034.152877] Lustre: Skipped 1 previous similar message [ 2037.563957] Lustre: server umount lustre-OST0001 complete [ 2039.266236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.266891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2039.267398] LustreError: 132628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.267404] LustreError: 132628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2039.270756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2039.281536] Lustre: Skipped 2 previous similar messages [ 2044.389482] LustreError: 133288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.393641] LustreError: 133288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2049.505536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2049.506794] LustreError: 135191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2049.507731] Lustre: Skipped 3 previous similar messages [ 2049.513353] LustreError: 135191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2053.087132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2053.136264] Lustre: server umount lustre-OST0000 complete [ 2054.624300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2054.627268] Lustre: Skipped 2 previous similar messages [ 2060.371863] Lustre: server umount lustre-MDT0000 complete [ 2061.503115] LustreError: 131060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535264 with bad export cookie 1458746876418612505 [ 2061.506169] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2061.506494] LustreError: 131060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2061.610033] Lustre: server umount lustre-MDT0001 complete [ 2067.329069] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2068.374346] Key type lgssc unregistered [ 2068.516404] LNet: 136871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.519097] LNetError: 136871:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.536224] LNet: Removed LNI 192.168.203.115@tcp [ 2068.818818] Key type .llcrypt unregistered [ 2068.819960] Key type ._llcrypt unregistered [ 2080.758713] Key type ._llcrypt registered [ 2080.759684] Key type .llcrypt registered [ 2080.794715] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 2086.373550] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2086.706733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2086.747644] alg: No test for adler32 (adler32-zlib) [ 2087.590066] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 2087.659297] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2089.231125] Key type lgssc registered [ 2089.540412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2099.711212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2099.717028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.798705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2100.809836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2100.841174] Lustre: lustre-MDT0000: new disk, initializing [ 2100.862177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.869187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2102.006751] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2105.902774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.942646] Lustre: 140965:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2105.964841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2105.967710] Lustre: Skipped 1 previous similar message [ 2106.001915] Lustre: lustre-MDT0001: new disk, initializing [ 2106.029607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2106.040883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2106.044487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2107.308082] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2110.425876] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2111.583354] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2113.940611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2114.047091] Lustre: lustre-OST0000: new disk, initializing [ 2114.050138] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2114.055230] Lustre: 142707:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2114.092209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.031892] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2119.149699] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2119.162086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2119.166870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2119.203681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2121.116979] Lustre: Failing over lustre-OST0000 [ 2121.188441] Lustre: server umount lustre-OST0000 complete [ 2134.092788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2134.195539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.201440] Lustre: Skipped 1 previous similar message [ 2134.205072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2134.211419] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2136.040702] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2136.117627] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2137.382815] Lustre: Failing over lustre-MDT0000 [ 2137.478425] Lustre: server umount lustre-MDT0000 complete [ 2139.617886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.623891] Lustre: Skipped 2 previous similar messages [ 2140.639221] Lustre: 139195:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780535327/real 1780535327] req@00000000c7ab1504 x1867026572272384/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780535343 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2144.736217] LustreError: 141750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.741260] LustreError: 141750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2145.759206] Lustre: 139192:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780535332/real 1780535332] req@000000003ba665d0 x1867026572274176/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780535348 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2146.271377] LustreError: 140973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.857244] LustreError: 140971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.865439] LustreError: 140971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2150.419363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2150.462503] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2150.466600] Lustre: 140968:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2150.541393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2150.560562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.591180] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2151.391134] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2151.899662] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2156.003848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2156.011328] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2156.034111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2161.120947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2161.123140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2161.124059] LustreError: Skipped 1 previous similar message [ 2161.127847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2161.131575] Lustre: Skipped 1 previous similar message [ 2163.017547] Lustre: server umount lustre-OST0000 complete [ 2166.240680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2166.240862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2166.242979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2166.245536] Lustre: Skipped 2 previous similar messages [ 2166.248465] Lustre: Skipped 1 previous similar message [ 2170.261812] Lustre: server umount lustre-MDT0000 complete [ 2171.361068] LustreError: 140976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.366057] LustreError: 140976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2171.495098] LustreError: 140957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535374 with bad export cookie 6113513723737259147 [ 2171.499056] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2171.499092] LustreError: 140957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2171.612591] Lustre: server umount lustre-MDT0001 complete [ 2175.028742] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2176.010817] Key type lgssc unregistered [ 2176.143372] LNet: 146198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2176.145663] LNetError: 146198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2176.152296] LNet: Removed LNI 192.168.203.115@tcp [ 2176.451105] Key type .llcrypt unregistered [ 2176.452458] Key type ._llcrypt unregistered [ 2196.567381] Key type ._llcrypt registered [ 2196.568446] Key type .llcrypt registered [ 2196.608600] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2197.090437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2197.151474] alg: No test for adler32 (adler32-zlib) [ 2198.030572] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 2198.132208] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2199.727205] Key type lgssc registered [ 2200.101244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2202.752517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2202.759257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.914113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.285034] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2208.324817] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2210.308062] LustreError: 147728:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2210.311843] LustreError: 147728:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2210.315553] LustreError: 147728:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2210.435460] Lustre: server umount lustre-MDT0000 complete [ 2215.151484] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 2217.913688] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2228.312093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.399112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2228.412217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2228.452639] Lustre: lustre-MDT0000: new disk, initializing [ 2228.479501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2228.486653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2229.748744] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2233.831986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.878209] Lustre: 150880:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2233.901860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2233.903668] Lustre: Skipped 1 previous similar message [ 2233.937923] Lustre: lustre-MDT0001: new disk, initializing [ 2233.971290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2233.987764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2233.992952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2235.296574] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2238.363648] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2239.417661] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2241.474407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2241.565607] Lustre: lustre-OST0000: new disk, initializing [ 2241.567130] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2241.569450] Lustre: 152624:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2241.594887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2243.240895] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2244.595252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2244.597927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2244.624966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2246.059149] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2247.762272] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2248.316474] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:10:51 (1780535451) [ 2311.139151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.139935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.143344] Lustre: Skipped 2 previous similar messages [ 2311.143493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2311.145087] Lustre: Skipped 2 previous similar messages [ 2313.638486] Lustre: server umount lustre-MDT0000 complete [ 2314.854616] LustreError: 151659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535517 with bad export cookie 9826211950923554421 [ 2314.855732] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2314.858467] LustreError: 151659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2316.255429] LustreError: 152013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.255483] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.261975] LustreError: 152013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2316.267387] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2319.263419] LustreError: 150887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.263469] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2320.995337] Lustre: server umount lustre-MDT0001 complete [ 2328.398319] Lustre: server umount lustre-OST0000 complete [ 2331.386659] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 2334.454425] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2345.071991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.149174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2345.157426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2345.190155] Lustre: lustre-MDT0000: new disk, initializing [ 2345.208120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2345.213036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2346.341920] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2350.127512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.161974] Lustre: 157105:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2350.183791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2350.185945] Lustre: Skipped 1 previous similar message [ 2350.217558] Lustre: lustre-MDT0001: new disk, initializing [ 2350.253066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2350.255772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2351.348408] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2354.179532] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2355.224027] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2357.337698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2357.425242] Lustre: 158844:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2357.447412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2357.449841] Lustre: Skipped 1 previous similar message [ 2359.170463] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2360.499668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2360.504359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2360.527855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2362.017562] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2363.218141] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2363.281299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2364.465364] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2364.537629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2365.922337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2365.923078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2365.923440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2365.927254] Lustre: Skipped 1 previous similar message [ 2371.041638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2371.043230] Lustre: Skipped 2 previous similar messages [ 2371.213445] Lustre: server umount lustre-OST0000 complete [ 2376.160662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.160916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2376.166865] LustreError: Skipped 1 previous similar message [ 2378.448823] Lustre: server umount lustre-MDT0000 complete [ 2379.658200] LustreError: 157096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535582 with bad export cookie 9826211950923713237 [ 2379.661467] LustreError: 157096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2379.661665] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2379.772634] Lustre: server umount lustre-MDT0001 complete [ 2382.388393] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:13:05 (1780535585) [ 2387.415367] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 2390.499379] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2401.685541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2401.772203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2401.782456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2401.785563] Lustre: Skipped 1 previous similar message [ 2401.826984] Lustre: lustre-MDT0000: new disk, initializing [ 2401.828555] Lustre: Skipped 1 previous similar message [ 2401.852341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2401.858074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2403.122463] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2407.166688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.210405] Lustre: 163415:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2407.311666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2408.624270] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2411.669859] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2412.861298] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2415.085164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2415.217912] Lustre: 165164:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2417.132789] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2418.936614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2418.971213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2420.221788] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2424.289339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2424.290217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.291905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.291909] Lustre: Skipped 2 previous similar messages [ 2424.292549] LustreError: Skipped 1 previous similar message [ 2424.295723] Lustre: Skipped 2 previous similar messages [ 2428.178118] Lustre: server umount lustre-OST0000 complete [ 2435.551802] Lustre: server umount lustre-MDT0000 complete [ 2436.730162] LustreError: 163406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535639 with bad export cookie 9826211950923714105 [ 2436.733310] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2436.734636] LustreError: 163406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2436.846351] Lustre: server umount lustre-MDT0001 complete [ 2439.538838] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:14:03 (1780535643) [ 2444.694347] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 2447.826735] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2459.254141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2459.342631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2459.353878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2459.356696] Lustre: Skipped 3 previous similar messages [ 2459.392968] Lustre: lustre-MDT0000: new disk, initializing [ 2459.394344] Lustre: Skipped 2 previous similar messages [ 2459.419357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2459.421115] Lustre: Skipped 2 previous similar messages [ 2459.425914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2459.427963] Lustre: Skipped 2 previous similar messages [ 2460.646629] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2464.619427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.665456] Lustre: 169335:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2464.765300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2466.020839] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2469.009543] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2470.191866] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2472.321047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2472.428196] Lustre: 171084:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2474.237014] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2477.254705] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2479.639921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2484.706815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.707782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2484.708492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2484.708496] Lustre: Skipped 5 previous similar messages [ 2484.710292] Lustre: Skipped 3 previous similar messages [ 2484.712166] LustreError: Skipped 1 previous similar message [ 2486.544640] Lustre: server umount lustre-OST0000 complete [ 2489.815783] Lustre: server umount lustre-MDT0000 complete [ 2489.826362] LustreError: 170635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.831518] LustreError: 170635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2491.026121] LustreError: 170116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535693 with bad export cookie 9826211950923715176 [ 2491.029260] LustreError: 170116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2491.029459] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2491.138219] Lustre: server umount lustre-MDT0001 complete [ 2494.537111] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2495.568469] Key type lgssc unregistered [ 2495.687361] LNet: 172811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2495.690228] LNetError: 172811:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2495.700306] LNet: Removed LNI 192.168.203.115@tcp [ 2495.985961] Key type .llcrypt unregistered [ 2495.987839] Key type ._llcrypt unregistered [ 2504.612838] Key type ._llcrypt registered [ 2504.613956] Key type .llcrypt registered [ 2504.651965] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2505.113546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2505.123224] alg: No test for adler32 (adler32-zlib) [ 2505.983152] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 2506.064733] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2507.647122] Key type lgssc registered [ 2508.007301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2510.487687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2510.495037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2511.659272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.938173] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2516.015327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2516.141447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2517.417470] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2519.174598] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2520.488801] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2522.734554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2522.857371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.704108] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2526.401055] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2537.190602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2556.896067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2556.899236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.904962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2557.921481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2557.921536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2557.922088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.923411] Lustre: Skipped 1 previous similar message [ 2559.631307] Lustre: server umount lustre-OST0000 complete [ 2562.527773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2562.530023] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.533890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.160486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.162395] Lustre: Skipped 3 previous similar messages [ 2573.280149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.282139] Lustre: Skipped 1 previous similar message [ 2575.327113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2575.382629] Lustre: server umount lustre-MDT0000 complete [ 2576.587235] LustreError: 174167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535779 with bad export cookie 5680185164959353397 [ 2576.590529] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.594061] LustreError: 174167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2576.700230] Lustre: server umount lustre-MDT0001 complete [ 2580.117272] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2581.111466] Key type lgssc unregistered [ 2581.226373] LNet: 177388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2581.230349] LNetError: 177388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2581.241318] LNet: Removed LNI 192.168.203.115@tcp [ 2581.514242] Key type .llcrypt unregistered [ 2581.515284] Key type ._llcrypt unregistered [ 2590.127196] Key type ._llcrypt registered [ 2590.128300] Key type .llcrypt registered [ 2590.170207] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2590.561357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2590.567493] alg: No test for adler32 (adler32-zlib) [ 2591.415943] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 2591.501044] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2593.079133] Key type lgssc registered [ 2593.428263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2595.896241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2595.901666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.051594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.263445] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2601.038176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.154516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2602.399986] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2604.124499] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2605.276678] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2607.382629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2607.487626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.224987] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2610.912141] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2621.862262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2631.135978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2631.139273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.144791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2631.660344] Lustre: lustre-OST0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 2632.163865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2632.166324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2635.149628] Lustre: server umount lustre-OST0000 complete [ 2636.902508] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90bec41bc000.ost_server_uuid 50 [ 2637.481245] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2647.453551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.454533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2647.458021] Lustre: Skipped 1 previous similar message [ 2647.460130] Lustre: Skipped 2 previous similar messages [ 2652.640054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.642340] Lustre: Skipped 1 previous similar message [ 2653.593391] Lustre: server umount lustre-MDT0000 complete [ 2654.809184] LustreError: 180383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535857 with bad export cookie 16529101531131435565 [ 2654.812650] LustreError: 180383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2654.812948] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2654.920909] Lustre: server umount lustre-MDT0001 complete [ 2657.560858] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:17:41 (1780535861) [ 2659.767950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.936807] LustreError: 182247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.942384] LustreError: 182247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2659.964664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.273630] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2663.921226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.042885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2665.300653] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2666.967301] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2668.100937] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2670.192646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2671.335206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2672.185621] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2673.795457] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2676.000920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2676.072773] Lustre: lustre-OST0001: new disk, initializing [ 2676.074260] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2676.076341] Lustre: 184749:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2676.104382] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2676.105995] Lustre: Skipped 1 previous similar message [ 2677.946504] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2680.320500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2680.325829] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2680.352083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2680.997305] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2681.823919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2681.826509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.830988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2683.871940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2690.529201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2690.532470] Lustre: Skipped 5 previous similar messages [ 2696.159104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2696.195540] Lustre: server umount lustre-OST0000 complete [ 2696.672185] LustreError: 183845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.078959] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2698.146888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2699.428275] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2699.504835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2700.773694] LustreError: 184757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.777620] LustreError: 184757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2705.891985] LustreError: 183843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.896752] LustreError: 183843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2711.009521] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2711.009893] LustreError: 183844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.010250] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.010257] Lustre: Skipped 1 previous similar message [ 2711.013025] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2711.015332] LustreError: 183844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2711.018835] Lustre: Skipped 4 previous similar messages [ 2715.601064] Lustre: server umount lustre-OST0001 complete [ 2720.224138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2720.228120] LustreError: Skipped 1 previous similar message [ 2720.230499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.237243] Lustre: Skipped 1 previous similar message [ 2722.978219] Lustre: server umount lustre-MDT0000 complete [ 2724.343419] LustreError: 182272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535927 with bad export cookie 16529101531131436622 [ 2724.346594] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2724.346947] LustreError: 182272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2724.458434] Lustre: server umount lustre-MDT0001 complete [ 2732.769554] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:18:56 (1780535936) [ 2735.012580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2735.105213] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2735.116340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2735.183169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2736.492941] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2739.517902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2739.549542] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2739.561282] Lustre: 188022:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2740.851671] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2742.473327] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2743.551711] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2745.645459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2745.705103] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2747.513688] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2749.234508] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2749.971111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2751.455264] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2751.478927] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2753.403443] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2755.162616] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2756.580238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.584871] Lustre: Skipped 1 previous similar message [ 2756.587317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2756.589413] Lustre: Skipped 4 previous similar messages [ 2762.104891] Lustre: server umount lustre-OST0000 complete [ 2764.263087] LustreError: 189644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.273507] LustreError: 189644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2774.499084] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2778.831316] Lustre: server umount lustre-OST0001 complete [ 2786.213920] Lustre: server umount lustre-MDT0000 complete [ 2787.686544] LustreError: 188751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535990 with bad export cookie 16529101531131438015 [ 2787.690655] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2787.691931] LustreError: 188751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2787.826518] Lustre: server umount lustre-MDT0001 complete [ 2795.577648] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:19:59 (1780535999) [ 2800.684142] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 2803.558920] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2816.676533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.764043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2816.774750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2816.815282] Lustre: lustre-MDT0000: new disk, initializing [ 2816.839125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2816.841346] Lustre: Skipped 3 previous similar messages [ 2816.846925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2818.235302] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2823.482060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.554990] Lustre: 195595:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2823.599834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2823.604354] Lustre: Skipped 1 previous similar message [ 2823.660330] Lustre: lustre-MDT0001: new disk, initializing [ 2823.712160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2823.717543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2825.239660] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2829.068575] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2830.750205] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2833.798797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2833.934973] Lustre: lustre-OST0000: new disk, initializing [ 2833.937790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2833.941749] Lustre: 197342:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2836.415871] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2839.049581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2839.057807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2839.099929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2840.338995] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2842.079349] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2842.195066] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2843.861169] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2843.922049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2849.251806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2849.252500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2849.254692] LustreError: Skipped 3 previous similar messages [ 2849.258600] Lustre: Skipped 5 previous similar messages [ 2849.260666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2849.264622] Lustre: Skipped 9 previous similar messages [ 2850.580167] Lustre: server umount lustre-OST0000 complete [ 2858.398178] Lustre: server umount lustre-MDT0000 complete [ 2859.487936] LustreError: 195607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2859.493422] LustreError: 195607:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2859.638260] LustreError: 195589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536062 with bad export cookie 16529101531131439534 [ 2859.641799] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2859.644597] LustreError: 195589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2859.759419] Lustre: server umount lustre-MDT0001 complete [ 2862.511138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.925186] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2866.529220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.933961] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2869.707123] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2870.869871] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2873.104408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2874.992627] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2876.715640] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2878.036770] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2888.340558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2889.605976] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2889.680132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2896.409301] Lustre: server umount lustre-OST0000 complete [ 2898.458368] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2899.763726] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2923.124936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2923.225065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.227117] Lustre: Skipped 5 previous similar messages [ 2924.957648] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2926.596409] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2927.110237] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2934.152383] Lustre: server umount lustre-OST0000 complete [ 2938.338360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2938.338512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.343334] LustreError: Skipped 3 previous similar messages [ 2938.351528] Lustre: Skipped 5 previous similar messages [ 2941.457971] Lustre: server umount lustre-MDT0000 complete [ 2942.432180] LustreError: 199413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.436838] LustreError: 199413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2942.680172] LustreError: 201028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536145 with bad export cookie 16529101531131440416 [ 2942.682751] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2942.685911] LustreError: 201028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2942.788483] Lustre: server umount lustre-MDT0001 complete [ 2945.419928] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:22:28 (1780536148) [ 2947.689549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2949.281244] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2952.114613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2953.407171] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2955.114289] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2956.257774] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2958.365831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2960.205458] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2961.867583] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2963.143408] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2974.428718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec [ 2975.714470] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2975.827742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2978.074074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2978.137949] Lustre: lustre-OST0001: new disk, initializing [ 2978.139530] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2978.141732] Lustre: 208004:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2980.021871] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2982.947358] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2984.183544] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2988.031085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2988.034055] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2988.058071] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2988.317074] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2989.563871] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2989.628926] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2993.128551] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2993.132698] Lustre: Skipped 9 previous similar messages [ 2996.477409] Lustre: server umount lustre-OST0001 complete [ 2998.239412] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2998.316758] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2999.726796] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2999.791061] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3008.483784] LustreError: 206522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.489250] LustreError: 206522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3023.683458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.525465] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3027.289342] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3027.797233] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3034.488509] Lustre: server umount lustre-OST0001 complete [ 3052.489770] Lustre: server umount lustre-OST0000 complete [ 3059.862968] Lustre: server umount lustre-MDT0000 complete [ 3060.989041] LustreError: 204909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536263 with bad export cookie 16529101531131441340 [ 3060.991309] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3060.991915] LustreError: 204909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3067.179453] Lustre: server umount lustre-MDT0001 complete [ 3075.178833] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:24:38 (1780536278) [ 3077.221152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3077.306960] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3077.317509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3077.382909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3077.384726] Lustre: Skipped 5 previous similar messages [ 3078.538695] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3081.189612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.227500] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3081.236188] Lustre: 212974:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3082.554178] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3084.177357] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3085.304644] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3087.376239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3087.458787] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3089.262272] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3090.940378] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3099.853530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3099.876422] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3101.656588] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3103.414460] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3105.986918] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3107.305302] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3107.380178] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3108.680705] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3108.769183] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3109.994203] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90bec3835000.ost_server_uuid 50 [ 3110.493379] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90bec3835000.ost_server_uuid in FULL state after 0 sec [ 3110.855097] Lustre: Permanently deactivating lustre-OST0001 [ 3110.860579] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3111.736326] Lustre: Permanently reactivating lustre-OST0001 [ 3114.977427] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3114.978029] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3114.979935] LustreError: Skipped 5 previous similar messages [ 3114.984610] Lustre: Skipped 6 previous similar messages [ 3118.779606] Lustre: server umount lustre-OST0001 complete [ 3126.159404] Lustre: server umount lustre-OST0000 complete [ 3133.528290] Lustre: server umount lustre-MDT0000 complete [ 3134.818953] LustreError: 212964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536337 with bad export cookie 16529101531131442607 [ 3134.822243] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.824876] LustreError: 212964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3134.965450] Lustre: server umount lustre-MDT0001 complete [ 3146.053275] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:25:49 (1780536349) [ 3149.368547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.516166] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3149.527486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3149.534359] Lustre: Skipped 5 previous similar messages [ 3151.109176] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3154.394418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.428781] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3154.440657] Lustre: 219080:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3156.066883] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3158.325396] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3159.533938] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3162.619863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3162.758940] Lustre: Permanently deactivating lustre-OST0000 [ 3162.767226] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3164.727143] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3164.730069] LustreError: 220988:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 3165.049025] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3167.061588] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3170.264961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3172.568290] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3172.719695] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3174.855228] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3175.484206] Lustre: Permanently reactivating lustre-OST0000 [ 3182.602117] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3182.607285] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3182.615075] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3182.635780] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3182.648703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3193.360032] Lustre: server umount lustre-OST0001 complete [ 3198.437786] LustreError: 220699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.445792] LustreError: 220699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 3209.183130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3209.263775] Lustre: server umount lustre-OST0000 complete [ 3217.401560] Lustre: server umount lustre-MDT0000 complete [ 3218.938955] LustreError: 219803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536421 with bad export cookie 16529101531131444238 [ 3218.942831] LustreError: 219803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3218.943228] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3219.076194] Lustre: server umount lustre-MDT0001 complete [ 3222.937389] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3224.127250] Key type lgssc unregistered [ 3224.256481] LNet: 223489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3224.260956] LNetError: 223489:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3224.273314] LNet: Removed LNI 192.168.203.115@tcp [ 3224.588184] Key type .llcrypt unregistered [ 3224.589352] Key type ._llcrypt unregistered [ 3233.451031] Key type ._llcrypt registered [ 3233.453216] Key type .llcrypt registered [ 3233.496873] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3233.968608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3234.019549] alg: No test for adler32 (adler32-zlib) [ 3234.884760] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3234.986332] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3236.575256] Key type lgssc registered [ 3237.105857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3241.053155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3241.065154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3242.243447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3243.579613] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3246.526014] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.566347] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3246.577480] Lustre: 224907:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3246.583087] Lustre: Permanently deactivating lustre-MDT0001 [ 3246.589987] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3246.702880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3248.315409] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3250.315984] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3250.541212] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3250.543219] LustreError: 224948:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3250.546766] LustreError: 224948:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3250.549680] LustreError: 224948:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 3251.504970] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3254.211044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3254.390921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3256.391424] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3256.423418] LustreError: 226533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3256.428320] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 3256.430740] LustreError: 226533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3258.160953] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3260.742037] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3260.816278] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3262.679924] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3264.511461] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3266.022222] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3268.081379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3268.082090] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3270.535516] Lustre: Permanently reactivating lustre-MDT0001 [ 3270.540886] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3270.542865] Lustre: Skipped 2 previous similar messages [ 3274.475131] Lustre: lustre-MDT0001: Client 2aa20b0d-130a-4389-9a6c-c3a47988d986 (at 192.168.203.15@tcp) reconnecting [ 3276.104754] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3276.836590] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3278.946758] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3279.021813] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3280.742256] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3280.872756] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3281.995752] Lustre: Permanently deactivating lustre-MDT0001 [ 3282.007776] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3282.010394] Lustre: Skipped 2 previous similar messages [ 3287.577885] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3292.129580] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3292.131487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3292.135137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3296.154437] Lustre: server umount lustre-MDT0000 complete [ 3297.760691] LustreError: 228013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536500 with bad export cookie 831738039161286912 [ 3297.768469] LustreError: 224917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3297.768905] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.769406] LustreError: 228013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3297.778702] LustreError: 224917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3297.898176] Lustre: server umount lustre-MDT0001 complete [ 3305.795811] Lustre: server umount lustre-OST0000 complete [ 3310.265543] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:28:33 (1780536513) [ 3314.745602] Lustre: server umount lustre-OST0001 complete [ 3317.099420] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 3320.323608] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3334.934952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.068341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3335.074416] Lustre: Skipped 2 previous similar messages [ 3335.088282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3335.135841] Lustre: lustre-MDT0000: new disk, initializing [ 3335.167138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.175607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3336.830335] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3341.626378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.673295] Lustre: 232563:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3341.704033] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3341.706411] Lustre: Skipped 1 previous similar message [ 3341.747468] Lustre: lustre-MDT0001: new disk, initializing [ 3341.800953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3341.804597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3343.195234] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3346.387584] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3347.742268] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3350.388642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3350.494408] Lustre: lustre-OST0000: new disk, initializing [ 3350.496261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3350.498908] Lustre: 234309:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3350.526436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3350.529921] Lustre: Skipped 1 previous similar message [ 3352.452080] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3355.976604] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3359.223817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3359.229342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3359.254873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3361.967345] LustreError: 232572:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3366.144400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3366.204389] Lustre: lustre-OST0001: new disk, initializing [ 3366.206635] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3366.209920] Lustre: 235315:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3366.439733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3366.444768] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3366.481760] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3368.396484] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3371.452137] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3372.063063] LustreError: 232572:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3372.070389] LustreError: 234859:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3373.060250] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3382.159106] LustreError: 234859:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3386.853357] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3386.855734] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.858707] LustreError: Skipped 1 previous similar message [ 3386.865862] Lustre: Skipped 3 previous similar messages [ 3386.870364] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3386.875133] Lustre: Skipped 4 previous similar messages [ 3388.881871] Lustre: server umount lustre-OST0001 complete [ 3391.975217] LustreError: 234917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3391.985710] LustreError: 234917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3397.093672] LustreError: 234293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3397.098677] LustreError: 234293:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3402.216354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.216739] LustreError: 234919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.219283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3402.219301] Lustre: Skipped 1 previous similar message [ 3402.224331] Lustre: Skipped 1 previous similar message [ 3402.241366] LustreError: 234919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3407.333581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3407.338513] Lustre: Skipped 2 previous similar messages [ 3407.598277] Lustre: server umount lustre-OST0000 complete [ 3412.450535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3412.451983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3412.458234] Lustre: Skipped 1 previous similar message [ 3412.464434] Lustre: Skipped 1 previous similar message [ 3415.451347] Lustre: server umount lustre-MDT0000 complete [ 3416.794228] LustreError: 232556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536619 with bad export cookie 831738039161289950 [ 3416.797906] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3416.798719] LustreError: 232556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3431.391167] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3431.538823] Lustre: server umount lustre-MDT0001 complete [ 3435.199267] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3436.508964] Key type lgssc unregistered [ 3436.646665] LNet: 237216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3436.651080] LNetError: 237216:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3436.659630] LNet: Removed LNI 192.168.203.115@tcp [ 3437.097460] Key type .llcrypt unregistered [ 3437.099787] Key type ._llcrypt unregistered [ 3454.364335] Key type ._llcrypt registered [ 3454.365600] Key type .llcrypt registered [ 3454.407971] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3454.901253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3454.906691] alg: No test for adler32 (adler32-zlib) [ 3455.752286] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3455.833647] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3457.415172] Key type lgssc registered [ 3457.769479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3461.095940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3461.112919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3462.239490] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3462.251462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3462.328646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3463.746770] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3466.779784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3466.834309] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3466.852491] Lustre: 239386:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3466.960689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3468.293351] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3470.590451] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3472.182684] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3475.175981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3475.250205] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3475.307806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3476.956883] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3479.363870] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3479.570384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3525.601920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3525.605315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.606241] LustreError: Skipped 1 previous similar message [ 3525.614928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3525.620361] Lustre: Skipped 1 previous similar message [ 3528.940213] Lustre: server umount lustre-OST0000 complete [ 3530.641777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3535.161947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3535.297743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3537.250837] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3539.246543] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3545.576933] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3545.585190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3545.585464] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3545.597359] Lustre: Skipped 1 previous similar message [ 3576.291577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3576.294811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3576.296237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3576.301520] Lustre: Skipped 2 previous similar messages [ 3576.306577] LustreError: Skipped 1 previous similar message [ 3579.226747] Lustre: server umount lustre-OST0000 complete [ 3581.409813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3581.410803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3581.415375] Lustre: Skipped 1 previous similar message [ 3581.419216] Lustre: Skipped 2 previous similar messages [ 3586.528866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3586.532041] Lustre: Skipped 1 previous similar message [ 3591.648456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3591.651668] Lustre: Skipped 1 previous similar message [ 3595.231224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3595.315560] Lustre: server umount lustre-MDT0000 complete [ 3596.795092] LustreError: 241742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536799 with bad export cookie 6669460064311860115 [ 3596.798444] LustreError: 241742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3596.798753] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3596.911237] Lustre: server umount lustre-MDT0001 complete [ 3600.676272] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3601.819568] Key type lgssc unregistered [ 3601.956681] LNet: 244335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3601.960944] LNetError: 244335:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3601.969479] LNet: Removed LNI 192.168.203.115@tcp [ 3602.280598] Key type .llcrypt unregistered [ 3602.281691] Key type ._llcrypt unregistered [ 3612.368863] Key type ._llcrypt registered [ 3612.370796] Key type .llcrypt registered [ 3612.409054] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3612.829847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3612.838140] alg: No test for adler32 (adler32-zlib) [ 3613.713170] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3613.819728] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3615.415174] Key type lgssc registered [ 3615.915973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3619.427299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3619.438050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.596730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3622.135918] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3625.301966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3625.512437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3627.171156] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3629.352753] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3630.584276] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3633.947221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3634.081355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3636.435524] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3638.725818] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3646.378270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3661.794771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3661.795630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3661.798146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3661.802562] Lustre: Skipped 1 previous similar message [ 3661.807072] LustreError: Skipped 1 previous similar message [ 3664.365601] Lustre: server umount lustre-OST0000 complete [ 3666.400381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3666.405939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.415660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3666.420018] Lustre: Skipped 1 previous similar message [ 3672.032334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3672.036471] Lustre: Skipped 2 previous similar messages [ 3672.281603] Lustre: server umount lustre-MDT0000 complete [ 3673.636221] LustreError: 248675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536876 with bad export cookie 16065537347611364831 [ 3673.644073] LustreError: 248675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3673.644379] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3673.814719] Lustre: server umount lustre-MDT0001 complete [ 3678.315563] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3679.305968] Key type lgssc unregistered [ 3679.433758] LNet: 249698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3679.439230] LNetError: 249698:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3679.452516] LNet: Removed LNI 192.168.203.115@tcp [ 3679.898597] Key type .llcrypt unregistered [ 3679.901068] Key type ._llcrypt unregistered [ 3687.366247] Key type ._llcrypt registered [ 3687.367354] Key type .llcrypt registered [ 3687.399432] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3687.687147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3687.694371] alg: No test for adler32 (adler32-zlib) [ 3688.582831] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3688.685889] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3690.279172] Key type lgssc registered [ 3690.713499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3693.586712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3693.592597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3694.809264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3696.602625] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3699.812079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3699.921391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3701.165880] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3703.592194] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3704.918464] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3707.422800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3707.603214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3710.041899] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3712.443346] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3719.926570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3730.400926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3730.402140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.404130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3730.407577] Lustre: Skipped 1 previous similar message [ 3730.411078] Lustre: Skipped 1 previous similar message [ 3733.604792] Lustre: server umount lustre-OST0000 complete [ 3735.521435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.531626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3735.535823] Lustre: Skipped 1 previous similar message [ 3740.640736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3740.643888] Lustre: Skipped 1 previous similar message [ 3741.471397] Lustre: server umount lustre-MDT0000 complete [ 3743.095231] LustreError: 250862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536945 with bad export cookie 14667346286938919454 [ 3743.099361] LustreError: 250862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3743.099657] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3743.234589] Lustre: server umount lustre-MDT0001 complete [ 3747.880455] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3748.933551] Key type lgssc unregistered [ 3749.061509] LNet: 254225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3749.065057] LNetError: 254225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3749.074317] LNet: Removed LNI 192.168.203.115@tcp [ 3749.431379] Key type .llcrypt unregistered [ 3749.432693] Key type ._llcrypt unregistered [ 3758.774543] Key type ._llcrypt registered [ 3758.775738] Key type .llcrypt registered [ 3758.818663] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3759.125259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3759.139269] alg: No test for adler32 (adler32-zlib) [ 3760.009412] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3760.111974] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3761.695177] Key type lgssc registered [ 3762.076305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3764.677098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3764.685260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3765.834720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3767.193320] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3770.745039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3770.858493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3772.084100] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3774.324505] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3775.576882] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3778.087220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3778.196433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3780.243285] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3781.907984] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3790.507051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3806.182221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3806.182580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3806.191922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3806.195747] LustreError: Skipped 1 previous similar message [ 3806.199501] Lustre: Skipped 1 previous similar message [ 3807.900319] Lustre: server umount lustre-OST0000 complete [ 3811.298158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.306425] Lustre: Skipped 1 previous similar message [ 3811.310465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3811.314564] Lustre: Skipped 1 previous similar message [ 3815.523406] Lustre: server umount lustre-MDT0000 complete [ 3816.416309] LustreError: 255603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3816.421691] LustreError: 255603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3816.802742] LustreError: 255585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537019 with bad export cookie 12898262604624705218 [ 3816.804713] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3816.806518] LustreError: 255585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3816.951392] Lustre: server umount lustre-MDT0001 complete [ 3820.348147] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3821.422184] Key type lgssc unregistered [ 3821.555469] LNet: 259574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3821.560254] LNetError: 259574:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3821.569365] LNet: Removed LNI 192.168.203.115@tcp [ 3821.887776] Key type .llcrypt unregistered [ 3821.889511] Key type ._llcrypt unregistered [ 3828.435497] Key type ._llcrypt registered [ 3828.436820] Key type .llcrypt registered [ 3828.477841] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3828.778555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3828.788656] alg: No test for adler32 (adler32-zlib) [ 3829.629668] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3829.709843] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3831.295189] Key type lgssc registered [ 3831.696298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3834.359773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3834.366085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3835.530248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3836.874634] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3839.709617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3839.831402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3840.952797] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3842.534323] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3843.594354] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3845.707754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3845.811158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3846.824159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3847.679314] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3849.383883] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3855.333131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.335632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3855.336755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3855.341128] Lustre: Skipped 1 previous similar message [ 3856.156560] Lustre: server umount lustre-OST0000 complete [ 3859.762109] Lustre: server umount lustre-MDT0000 complete [ 3860.450369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.450480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3860.456777] LustreError: 260782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3860.466407] LustreError: 260782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3861.569836] LustreError: 263393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537064 with bad export cookie 121360813426674598 [ 3861.575880] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3861.577568] LustreError: 263393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3861.732687] Lustre: server umount lustre-MDT0001 complete [ 3866.161469] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3867.535323] Key type lgssc unregistered [ 3867.714618] LNet: 264126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3867.720467] LNetError: 264126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3867.735754] LNet: Removed LNI 192.168.203.115@tcp [ 3868.206098] Key type .llcrypt unregistered [ 3868.207980] Key type ._llcrypt unregistered [ 3872.431575] Key type ._llcrypt registered [ 3872.432885] Key type .llcrypt registered [ 3872.489904] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3878.668661] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 3885.538420] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3885.820146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3885.827097] alg: No test for adler32 (adler32-zlib) [ 3886.665688] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3886.745302] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3888.327220] Key type lgssc registered [ 3888.892494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3904.713143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3904.724833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3905.832864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3905.842789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3905.876154] Lustre: lustre-MDT0000: new disk, initializing [ 3905.897272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3905.902966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3907.349612] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3912.120981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.187254] Lustre: 267602:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3912.224973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3912.228864] Lustre: Skipped 1 previous similar message [ 3912.274225] Lustre: lustre-MDT0001: new disk, initializing [ 3912.316990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3912.334091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3912.340822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3914.029402] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3917.515567] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3918.929918] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3921.644508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3921.739838] Lustre: lustre-OST0000: new disk, initializing [ 3921.741434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3921.743759] Lustre: 269343:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3921.772976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3922.230394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3922.233467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3922.267835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3923.687863] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3927.189287] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3929.123925] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3929.249622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3931.057219] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3931.161219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3932.642064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3932.644264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.647306] LustreError: Skipped 1 previous similar message [ 3932.657054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3932.662117] Lustre: Skipped 1 previous similar message [ 3937.761108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3937.763670] Lustre: Skipped 1 previous similar message [ 3938.144230] Lustre: server umount lustre-OST0000 complete [ 3942.882744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.884004] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3942.890835] Lustre: Skipped 2 previous similar messages [ 3942.902446] Lustre: Skipped 1 previous similar message [ 3945.756875] Lustre: server umount lustre-MDT0000 complete [ 3947.064177] LustreError: 270740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537149 with bad export cookie 11560692412699737353 [ 3947.066530] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3947.067646] LustreError: 270740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3947.196406] Lustre: server umount lustre-MDT0001 complete [ 3950.099306] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:39:13 (1780537153) [ 3952.453156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.652099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3954.106837] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3957.320412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3957.482545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3958.741275] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3960.902584] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3962.398595] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3964.762626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3967.033638] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3969.166619] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3983.333569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3983.333939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3983.336743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3983.338454] LustreError: Skipped 1 previous similar message [ 3983.345702] Lustre: Skipped 1 previous similar message [ 3988.449459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3988.451767] Lustre: Skipped 1 previous similar message [ 3988.948659] Lustre: server umount lustre-OST0000 complete [ 3993.058172] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3993.062917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3996.522262] Lustre: server umount lustre-MDT0000 complete [ 3998.168835] LustreError: 271592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537200 with bad export cookie 11560692412699738221 [ 3998.174333] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3998.176153] LustreError: 271592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3998.320080] Lustre: server umount lustre-MDT0001 complete [ 4002.644926] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 4003.706368] Key type lgssc unregistered [ 4003.837572] LNet: 274814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4003.842267] LNetError: 274814:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4003.852491] LNet: Removed LNI 192.168.203.115@tcp [ 4004.152975] Key type .llcrypt unregistered [ 4004.154032] Key type ._llcrypt unregistered [ 4013.896457] Key type ._llcrypt registered [ 4013.898142] Key type .llcrypt registered [ 4013.950991] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 4019.966126] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4020.291867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4020.331644] alg: No test for adler32 (adler32-zlib) [ 4021.181270] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 4021.277910] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 4022.855275] Key type lgssc registered [ 4023.275280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4042.459292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4042.464652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4043.595491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4043.610833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4043.655272] Lustre: lustre-MDT0000: new disk, initializing [ 4043.683919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.692943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4045.388749] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4050.548299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4050.594822] Lustre: 278578:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4050.622136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4050.624536] Lustre: Skipped 1 previous similar message [ 4050.664891] Lustre: lustre-MDT0001: new disk, initializing [ 4050.704239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4050.720268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4050.725155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4052.370099] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4055.957668] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4057.301544] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4060.406391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4060.546638] Lustre: lustre-OST2710: new disk, initializing [ 4060.550270] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4060.555277] Lustre: 280323:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4060.603591] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4062.908192] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4065.290142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4065.296265] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4065.345239] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4066.326581] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4069.425161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4069.530794] Lustre: lustre-OST03e8: new disk, initializing [ 4069.534508] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4069.540143] Lustre: 281272:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4069.592978] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4072.074440] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4073.243975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4073.250961] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4073.298414] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4074.494492] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4077.342292] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4077.449202] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4079.043264] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4079.118436] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4080.409116] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4080.480579] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4082.248897] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4082.369464] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4086.752693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4086.758025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.767101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4088.805821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4088.807632] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.810030] Lustre: Skipped 1 previous similar message [ 4088.818061] Lustre: Skipped 1 previous similar message [ 4090.040955] Lustre: server umount lustre-MDT0000 complete [ 4091.800977] LustreError: 278569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537294 with bad export cookie 17797002312872289337 [ 4091.808117] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4091.808612] LustreError: 278569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4091.982725] Lustre: server umount lustre-MDT0001 complete [ 4104.135132] Lustre: server umount lustre-OST2710 complete [ 4115.710819] Lustre: server umount lustre-OST03e8 complete [ 4123.767899] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 4127.807984] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4143.813298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.898662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4143.909518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4143.952412] Lustre: lustre-MDT0000: new disk, initializing [ 4143.982050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4143.989521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4145.610897] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4150.984216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.057378] Lustre: 286305:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4151.242318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4152.959222] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4156.945304] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4158.586653] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4161.715977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4161.852436] Lustre: lustre-OST0000: new disk, initializing [ 4161.855532] Lustre: Skipped 1 previous similar message [ 4161.858793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4161.862886] Lustre: Skipped 2 previous similar messages [ 4161.866913] Lustre: 288049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4161.911819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4161.916185] Lustre: Skipped 1 previous similar message [ 4164.407528] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4165.315584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4165.321341] Lustre: Skipped 1 previous similar message [ 4165.325596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4165.370108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4168.299712] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4170.149581] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4170.259089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4172.079830] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4172.179665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4175.843713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.845826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4175.852323] Lustre: Skipped 1 previous similar message [ 4175.857377] LustreError: Skipped 1 previous similar message [ 4175.861384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4175.867371] Lustre: Skipped 1 previous similar message [ 4179.173200] Lustre: server umount lustre-OST0000 complete [ 4180.961393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4180.962566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4180.969693] Lustre: Skipped 2 previous similar messages [ 4186.080948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4186.084855] Lustre: Skipped 2 previous similar messages [ 4195.295148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4195.297332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4195.305235] Lustre: Skipped 3 previous similar messages [ 4195.382433] Lustre: server umount lustre-MDT0000 complete [ 4197.182795] LustreError: 287084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537399 with bad export cookie 17797002312872291017 [ 4197.188909] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4197.192393] LustreError: 287084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4197.327873] Lustre: server umount lustre-MDT0001 complete [ 4200.825238] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:43:24 (1780537404) [ 4201.447567] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4202.272983] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:43:25 (1780537405) [ 4203.552833] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4214.698120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4214.817649] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4214.826159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4214.890765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4216.496252] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4220.067525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.101345] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4220.110307] Lustre: 291742:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4221.778888] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4238.884915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4239.006512] Lustre: Denying initial registration attempt from nid 192.168.203.115@tcp, specified as failover [ 4239.011993] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4239.018088] LustreError: 293162:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 4239.022662] LustreError: 293162:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 4239.027342] LustreError: 293162:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4239.064048] Lustre: server umount lustre-OST0000 complete [ 4239.066817] LustreError: 293162:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4245.474379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4245.475231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4245.482398] Lustre: Skipped 1 previous similar message [ 4248.236264] Lustre: server umount lustre-MDT0000 complete [ 4250.021164] LustreError: 291732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537452 with bad export cookie 17797002312872291899 [ 4250.028073] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4250.029872] LustreError: 291732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4250.166757] Lustre: server umount lustre-MDT0001 complete [ 4255.151170] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 4256.644643] Key type lgssc unregistered [ 4256.806532] LNet: 294264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4256.812268] LNetError: 294264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4256.823883] LNet: Removed LNI 192.168.203.115@tcp [ 4257.314281] Key type .llcrypt unregistered [ 4257.316734] Key type ._llcrypt unregistered [ 4261.535876] Key type ._llcrypt registered [ 4261.536972] Key type .llcrypt registered [ 4261.577922] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4261.886439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4261.896276] alg: No test for adler32 (adler32-zlib) [ 4262.770142] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 4262.867312] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 4264.455122] Key type lgssc registered [ 4265.079565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4293.498249] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4296.848546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4296.859028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4297.967211] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4297.980075] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4298.055925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4299.534789] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4303.480225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.533370] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4303.547552] Lustre: 296971:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4303.653657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4305.296411] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4307.438668] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4308.830501] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4324.965127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4325.046053] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4325.107736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4327.348220] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4329.216849] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4330.273793] Lustre: server umount lustre-OST0000 complete [ 4334.560502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4334.565315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4334.573994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4335.585139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4335.586215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.223428] Lustre: server umount lustre-MDT0000 complete [ 4340.019930] LustreError: 296965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537542 with bad export cookie 1733805775918691060 [ 4340.025686] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4340.027833] LustreError: 296965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4340.168102] Lustre: server umount lustre-MDT0001 complete [ 4344.484750] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 4345.933994] Key type lgssc unregistered [ 4346.057287] LNet: 300192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4346.060839] LNetError: 300192:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4346.072513] LNet: Removed LNI 192.168.203.115@tcp [ 4346.529219] Key type .llcrypt unregistered [ 4346.531728] Key type ._llcrypt unregistered [ 4357.963757] Key type ._llcrypt registered [ 4357.965850] Key type .llcrypt registered [ 4358.037428] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4358.582245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4358.589973] alg: No test for adler32 (adler32-zlib) [ 4359.493861] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 4359.611087] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 4361.207227] Key type lgssc registered [ 4361.709692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4365.055817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4365.062077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.239855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.588640] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4370.903872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4371.040875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4372.645480] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4374.987305] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4376.540053] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4381.347468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4381.534620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4383.929468] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4386.074083] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4401.125434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.127378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4401.139349] Lustre: Skipped 2 previous similar messages [ 4401.143519] Lustre: Skipped 1 previous similar message [ 4402.393928] Lustre: server umount lustre-MDT0000 complete [ 4402.662639] LustreError: 301565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4402.674672] LustreError: 301565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4403.829414] LustreError: 301548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537606 with bad export cookie 15547054987814791424 [ 4403.833563] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4403.837778] LustreError: 301548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4406.240233] LustreError: 301563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4406.240638] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4406.259649] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4406.263330] Lustre: Skipped 1 previous similar message [ 4407.775958] LustreError: 301564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.785110] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 4410.054635] Lustre: server umount lustre-MDT0001 complete [ 4413.114391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4417.196994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4428.768112] LustreError: 300880:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000677b0286 x1867028954636800/t0(0) o250->MGC192.168.203.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4428.941824] LustreError: 303175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4428.955184] LustreError: 303175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4428.984022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.027519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4430.607898] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4434.080602] LustreError: 304841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.086407] LustreError: 304841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4434.418656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.562385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4435.679722] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4437.846357] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4438.691058] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4443.625891] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4445.156782] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4448.842416] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4454.372850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.375504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4454.381395] Lustre: Skipped 1 previous similar message [ 4454.387903] Lustre: Skipped 2 previous similar messages [ 4456.427476] Lustre: server umount lustre-OST0000 complete [ 4459.491266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4459.501373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4459.505137] Lustre: Skipped 1 previous similar message [ 4469.728982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4469.733594] Lustre: Skipped 3 previous similar messages [ 4472.287192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4472.374444] Lustre: server umount lustre-MDT0000 complete [ 4474.059802] LustreError: 303187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537676 with bad export cookie 15547054987814798578 [ 4474.065619] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4474.067255] LustreError: 303187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4474.210498] Lustre: server umount lustre-MDT0001 complete [ 4479.044434] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 4480.257558] Key type lgssc unregistered [ 4480.421629] LNet: 307278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4480.428432] LNetError: 307278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4480.443565] LNet: Removed LNI 192.168.203.115@tcp [ 4480.936407] Key type .llcrypt unregistered [ 4480.938665] Key type ._llcrypt unregistered [ 4491.311806] Key type ._llcrypt registered [ 4491.313133] Key type .llcrypt registered [ 4491.351488] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4491.641421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4491.647852] alg: No test for adler32 (adler32-zlib) [ 4492.509313] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 4492.617400] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 4494.223181] Key type lgssc registered [ 4494.711115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4497.971107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4497.983824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4499.180865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4500.698368] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4504.304247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4504.464813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4506.000378] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4508.612349] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4510.478595] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4513.761200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4513.761966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4513.769440] Lustre: Skipped 1 previous similar message [ 4514.784786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4514.787869] Lustre: Skipped 2 previous similar messages [ 4518.061424] Lustre: server umount lustre-MDT0000 complete [ 4519.765896] LustreError: 308638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537722 with bad export cookie 12049081475035075262 [ 4519.771492] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4519.773379] LustreError: 308638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4519.929608] Lustre: server umount lustre-MDT0001 complete [ 4525.316117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4525.475968] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4525.495455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4525.596774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4527.130983] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4530.764334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4530.817465] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4530.838544] Lustre: 311017:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4530.943171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4532.682398] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4535.006261] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4536.673201] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4542.242574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4542.322349] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4542.327515] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4542.331513] LustreError: 312675:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 4542.334577] LustreError: 312675:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 4542.337478] LustreError: 312675:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4542.361837] Lustre: server umount lustre-OST0000 complete [ 4542.363543] LustreError: 312675:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4547.940356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4548.055689] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4548.160524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4550.686940] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4553.197926] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4554.807561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4556.810615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4556.864949] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4556.919158] Lustre: lustre-OST0001: new disk, initializing [ 4556.922379] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4556.927081] Lustre: 314103:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4556.974308] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4559.432778] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4561.695675] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4563.077377] Lustre: server umount lustre-OST0001 complete [ 4563.471858] LustreError: 313141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4563.486620] LustreError: 313141:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4564.960961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4564.961045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4564.966085] LustreError: Skipped 1 previous similar message [ 4564.975174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4564.982527] Lustre: Skipped 1 previous similar message [ 4568.548523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4568.551058] LustreError: 313142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4568.552905] Lustre: Skipped 1 previous similar message [ 4568.563083] LustreError: 313142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4573.666401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4573.667349] LustreError: 313141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4573.668592] Lustre: Skipped 1 previous similar message [ 4573.674347] LustreError: 313141:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4578.790681] LustreError: 314111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4578.800441] LustreError: 314111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4579.295134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4579.373989] Lustre: server umount lustre-OST0000 complete [ 4582.368486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4582.373407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.381520] Lustre: Skipped 1 previous similar message [ 4582.384879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4582.388930] Lustre: Skipped 2 previous similar messages [ 4595.167154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4595.256326] Lustre: server umount lustre-MDT0000 complete [ 4596.966435] LustreError: 311054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537799 with bad export cookie 12049081475035075801 [ 4596.972963] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4596.973425] LustreError: 311054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4597.093367] Lustre: server umount lustre-MDT0001 complete [ 4600.693808] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 4602.029879] Key type lgssc unregistered [ 4602.163578] LNet: 315935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4602.167750] LNetError: 315935:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4602.175507] LNet: Removed LNI 192.168.203.115@tcp [ 4602.639343] Key type .llcrypt unregistered [ 4602.641888] Key type ._llcrypt unregistered [ 4638.548961] Key type ._llcrypt registered [ 4638.550112] Key type .llcrypt registered [ 4638.587812] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 4644.899885] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4645.464242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4645.488677] alg: No test for adler32 (adler32-zlib) [ 4646.347113] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 4646.435565] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 4648.015119] Key type lgssc registered [ 4648.436964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4660.596584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4660.604285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4661.686982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4661.697679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4661.733038] Lustre: lustre-MDT0000: new disk, initializing [ 4661.754303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4661.760507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4662.947743] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4667.097092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4667.135774] Lustre: 320808:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4667.155440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4667.157064] Lustre: Skipped 1 previous similar message [ 4667.194159] Lustre: lustre-MDT0001: new disk, initializing [ 4667.225208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4667.236246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4667.239409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4668.360641] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4671.619847] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4672.974782] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4675.175391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4675.268246] Lustre: lustre-OST0000: new disk, initializing [ 4675.269927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4675.272549] Lustre: 322550:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4675.296507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4677.065210] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4678.965491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4678.968437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4678.996850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4680.020099] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4681.214317] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4681.272686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4682.480300] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4682.548909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4684.257859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4684.258180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.259912] LustreError: Skipped 1 previous similar message [ 4684.263825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4684.265823] Lustre: Skipped 1 previous similar message [ 4685.197394] Lustre: server umount lustre-OST0000 complete [ 4687.839763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4687.841813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.844775] Lustre: Skipped 1 previous similar message [ 4687.846297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4689.376097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4689.376823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.454713] Lustre: server umount lustre-MDT0000 complete [ 4693.954129] LustreError: 322549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537896 with bad export cookie 11071956530454908390 [ 4693.959070] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4693.961014] LustreError: 322549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4694.108675] Lustre: server umount lustre-MDT0001 complete [ 4697.434344] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:51:40 (1780537900) [ 4700.991586] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:51:44 (1780537904) [ 4705.090829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4705.293511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4706.636684] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4709.738767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4709.900726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4711.659561] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4714.185183] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4715.774444] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4718.690282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4718.858506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.315736] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4723.549889] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4737.232293] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4737.717591] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4738.195240] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4738.690746] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4739.279348] Lustre: Failing over lustre-MDT0000 [ 4739.436797] Lustre: server umount lustre-MDT0000 complete [ 4740.576043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4740.578510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4740.582450] LustreError: 325157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4740.586412] LustreError: 325157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4741.600070] LustreError: 325156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.605073] LustreError: 325156:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4744.164511] LustreError: 326363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4746.720028] LustreError: 325175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4746.724225] LustreError: 325175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4751.839970] LustreError: 325157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.844236] LustreError: 325157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4751.902055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.938145] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4752.019331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.046215] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4753.142857] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4754.398854] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4757.478757] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4757.486310] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4757.500714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4758.573059] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4759.053908] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4760.802926] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4761.391873] Lustre: Failing over lustre-MDT0000 [ 4761.590847] Lustre: server umount lustre-MDT0000 complete [ 4762.593267] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4762.593422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4762.594316] LustreError: 325156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.597629] Lustre: Skipped 3 previous similar messages [ 4774.360922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4774.413287] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4774.557288] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4774.880396] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4776.029486] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4780.005841] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4780.007947] Lustre: Skipped 2 previous similar messages [ 4780.019470] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4780.044427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4781.438221] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4782.028072] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4785.120256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4785.124298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4785.131568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4785.134915] Lustre: Skipped 1 previous similar message [ 4790.423425] Lustre: server umount lustre-MDT0000 complete [ 4791.948570] LustreError: 325139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537994 with bad export cookie 11071956530454911043 [ 4791.953412] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4791.953806] LustreError: 325139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4792.071667] Lustre: server umount lustre-MDT0001 complete [ 4803.621276] Lustre: server umount lustre-OST0000 complete [ 4810.512794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4810.730944] LustreError: 330943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.740048] LustreError: 330943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4810.765438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4810.769076] Lustre: Skipped 1 previous similar message [ 4812.331466] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4816.205960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4818.037596] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4820.265741] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4821.761384] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4824.005983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4826.052443] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4828.425065] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4835.374608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4840.419570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4840.423083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4840.426207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4840.432984] Lustre: Skipped 3 previous similar messages [ 4840.436958] Lustre: Skipped 3 previous similar messages [ 4842.450250] Lustre: server umount lustre-OST0000 complete [ 4845.651641] Lustre: server umount lustre-MDT0000 complete [ 4846.753222] LustreError: 330923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538049 with bad export cookie 11071956530454911785 [ 4846.756639] LustreError: 330923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4846.756778] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4846.869163] Lustre: server umount lustre-MDT0001 complete [ 4850.096052] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 4851.071511] Key type lgssc unregistered [ 4851.188504] LNet: 334140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4851.190847] LNetError: 334140:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4851.202454] LNet: Removed LNI 192.168.203.115@tcp [ 4851.476921] Key type .llcrypt unregistered [ 4851.478097] Key type ._llcrypt unregistered [ 4860.124978] Key type ._llcrypt registered [ 4860.126314] Key type .llcrypt registered [ 4860.162352] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4860.618102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4860.631439] alg: No test for adler32 (adler32-zlib) [ 4861.498204] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 4861.594223] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 4863.183115] Key type lgssc registered [ 4863.550723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4866.150317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4866.156970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.311149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4868.711719] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4871.716711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4871.860193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4873.221743] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4875.143487] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4876.432671] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4878.893735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4879.052144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4881.151685] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4882.891697] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4893.415682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4902.881565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4902.885612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4902.891357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4903.906762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4903.906962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4903.911969] Lustre: Skipped 1 previous similar message [ 4903.914436] Lustre: Skipped 1 previous similar message [ 4908.409424] Lustre: server umount lustre-MDT0000 complete [ 4909.026075] LustreError: 335640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4909.033861] LustreError: 335640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4909.995718] LustreError: 335597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538112 with bad export cookie 6442222016153116998 [ 4909.998469] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4910.001505] LustreError: 335597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4910.142801] Lustre: server umount lustre-MDT0001 complete [ 4922.035287] Lustre: server umount lustre-OST0000 complete [ 4928.243688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4928.472646] LustreError: 339012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4928.507959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4930.148335] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4933.601337] LustreError: 339013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4933.964932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4934.155512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4935.956371] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4939.197847] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4941.664824] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4944.885618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4945.055259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4947.339199] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4949.210148] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4953.253768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4963.810120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4963.811511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4963.814035] Lustre: Skipped 2 previous similar messages [ 4963.816615] Lustre: Skipped 1 previous similar message [ 4967.254228] Lustre: server umount lustre-MDT0000 complete [ 4968.519101] LustreError: 340628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538171 with bad export cookie 6442222016153118097 [ 4968.521699] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4968.524379] LustreError: 340628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4968.642401] Lustre: server umount lustre-MDT0001 complete [ 4980.117236] Lustre: server umount lustre-OST0000 complete [ 4985.193160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4985.340786] LustreError: 342585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4985.360528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4986.652289] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4989.559626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4990.984247] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4992.718121] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4993.932139] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4996.084319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4998.075854] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4999.851135] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5010.537151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5020.127771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5020.130284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5020.135363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5021.154566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5022.727523] Lustre: server umount lustre-OST0000 complete [ 5025.759830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5025.763380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5025.769019] Lustre: Skipped 1 previous similar message [ 5025.771507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5025.773496] Lustre: Skipped 2 previous similar messages [ 5030.113593] Lustre: server umount lustre-MDT0000 complete [ 5031.391916] LustreError: 342604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5031.397063] LustreError: 342604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5031.428445] LustreError: 344201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538234 with bad export cookie 6442222016153119105 [ 5031.431435] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5031.431639] LustreError: 344201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5031.538113] Lustre: server umount lustre-MDT0001 complete [ 5035.027280] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5036.047416] Key type lgssc unregistered [ 5036.166308] LNet: 345783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5036.168863] LNetError: 345783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5036.177334] LNet: Removed LNI 192.168.203.115@tcp [ 5036.492116] Key type .llcrypt unregistered [ 5036.493236] Key type ._llcrypt unregistered [ 5045.902890] Key type ._llcrypt registered [ 5045.903976] Key type .llcrypt registered [ 5045.945842] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5046.360297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5046.379897] alg: No test for adler32 (adler32-zlib) [ 5047.223466] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 5047.295458] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5048.879161] Key type lgssc registered [ 5049.196340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5051.509261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5051.514389] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5051.516740] LustreError: 347183:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5051.524024] LustreError: 347183:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5051.526085] LustreError: 347183:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5051.528067] LustreError: 347183:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5051.530294] LustreError: 347183:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5053.793166] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5053.795904] LustreError: 347580:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5053.805101] LustreError: 347580:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5053.808286] LustreError: 347580:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5053.812214] LustreError: 347580:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5053.815134] LustreError: 347580:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5059.737497] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5060.857310] Key type lgssc unregistered [ 5060.988457] LNet: 348565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5060.990934] LNetError: 348565:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5061.003450] LNet: Removed LNI 192.168.203.115@tcp [ 5061.316816] Key type .llcrypt unregistered [ 5061.318859] Key type ._llcrypt unregistered [ 5067.539138] Key type ._llcrypt registered [ 5067.540395] Key type .llcrypt registered [ 5067.581658] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 5073.773412] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5074.166374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5074.254107] alg: No test for adler32 (adler32-zlib) [ 5075.129872] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 5075.228539] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5076.823165] Key type lgssc registered [ 5077.191961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5088.185043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5088.193149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.319298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5089.333032] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5089.372769] Lustre: lustre-MDT0000: new disk, initializing [ 5089.402337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5089.410167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5090.651177] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5094.517540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5094.563293] Lustre: 351661:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5094.609399] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5094.611759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5094.613184] Lustre: Skipped 1 previous similar message [ 5094.647466] Lustre: lustre-MDT0001: new disk, initializing [ 5094.681978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5094.695581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5094.698406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5095.856044] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5098.996191] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5100.121402] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5102.490451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5102.605521] Lustre: lustre-OST0000: new disk, initializing [ 5102.607793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5102.612337] Lustre: 353407:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5102.648975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5104.460433] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5107.395347] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5108.734291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5108.737117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5108.757473] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5108.787419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5108.844738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5110.083971] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5110.152995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5113.824797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5113.825784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.828583] LustreError: Skipped 1 previous similar message [ 5113.832647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5113.835072] Lustre: Skipped 1 previous similar message [ 5116.880370] Lustre: server umount lustre-OST0000 complete [ 5118.944375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5118.944946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5118.947834] Lustre: Skipped 2 previous similar messages [ 5124.064588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5124.067173] Lustre: Skipped 2 previous similar messages [ 5124.243906] Lustre: server umount lustre-MDT0000 complete [ 5125.444494] LustreError: 352442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538328 with bad export cookie 6011807346945450262 [ 5125.448751] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5125.583832] Lustre: server umount lustre-MDT0001 complete [ 5128.183292] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5128.685933] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:58:52 (1780538332) [ 5130.711208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.893121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5132.182266] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5134.875284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5134.997183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5136.251258] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5137.880237] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5138.958623] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5141.136393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5143.118914] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5144.845744] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5147.050840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5147.122867] Lustre: lustre-OST0001: new disk, initializing [ 5147.124832] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5147.127823] Lustre: 358217:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5147.175542] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5147.179066] Lustre: Skipped 1 previous similar message [ 5148.953306] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5152.240640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5152.244051] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5152.261123] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5152.278198] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5157.348322] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5157.352951] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.354770] LustreError: Skipped 1 previous similar message [ 5157.366629] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5157.374058] Lustre: Skipped 1 previous similar message [ 5160.126499] Lustre: server umount lustre-OST0001 complete [ 5162.473231] LustreError: 358226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5162.479164] LustreError: 358226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5167.587100] LustreError: 357315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5167.592639] LustreError: 357315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5172.709428] LustreError: 358226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.709682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5172.712863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5172.712883] Lustre: Skipped 1 previous similar message [ 5172.721178] LustreError: 358226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5172.741929] Lustre: Skipped 2 previous similar messages [ 5177.832280] LustreError: 357316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5177.840690] LustreError: 357316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5178.836582] Lustre: server umount lustre-OST0000 complete [ 5181.407917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5181.410410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.423873] Lustre: server umount lustre-MDT0000 complete [ 5188.065105] LustreError: 355720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5188.076431] LustreError: 355720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5188.183592] LustreError: 358213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538390 with bad export cookie 6011807346945451144 [ 5188.187350] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5188.187955] LustreError: 358213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5194.359134] Lustre: server umount lustre-MDT0001 complete [ 5198.451253] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5199.672523] Key type lgssc unregistered [ 5199.842520] LNet: 360070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5199.847188] LNetError: 360070:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5199.855361] LNet: Removed LNI 192.168.203.115@tcp [ 5200.321835] Key type .llcrypt unregistered [ 5200.323823] Key type ._llcrypt unregistered [ 5220.568681] Key type ._llcrypt registered [ 5220.569975] Key type .llcrypt registered [ 5220.621953] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5221.067185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5221.084460] alg: No test for adler32 (adler32-zlib) [ 5221.936930] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 5222.022625] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5223.607194] Key type lgssc registered [ 5224.048235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5226.649460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5226.654816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5227.768971] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5227.786660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5227.882836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5229.622852] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5233.781036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5233.841537] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5233.864292] Lustre: 362829:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5234.015611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5235.517113] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5237.661020] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5238.905345] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5244.386663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5244.386931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.395892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5244.397788] Lustre: Skipped 1 previous similar message [ 5245.680861] Lustre: server umount lustre-MDT0000 complete [ 5247.107931] LustreError: 362820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538449 with bad export cookie 2435894019370830962 [ 5247.112794] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5247.113324] LustreError: 362820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5247.246917] Lustre: server umount lustre-MDT0001 complete [ 5250.838470] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:00:54 (1780538454) [ 5253.305488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5253.594379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5255.538602] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5258.722293] LustreError: 365115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5258.738797] LustreError: 365115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5259.401370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5259.608826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5261.330971] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5263.578090] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5264.924773] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5267.267729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5267.340556] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5267.408555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5269.187465] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5270.826245] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5280.072908] Lustre: Permanently deactivating lustre-OST0000 [ 5280.083109] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5288.448421] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5288.889041] LustreError: 367873:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5288.892017] LustreError: 367873:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 5288.893961] LustreError: 367873:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5289.601040] LustreError: 367921:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 5289.604703] LustreError: 367921:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5296.485640] Lustre: server umount lustre-OST0000 complete [ 5297.527676] LustreError: 368170:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 5297.530024] LustreError: 368170:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5300.703739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5300.705867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5300.709167] Lustre: Skipped 1 previous similar message [ 5300.710773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5302.239980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5302.240627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.361034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5307.364965] Lustre: Skipped 2 previous similar messages [ 5307.397366] Lustre: server umount lustre-MDT0000 complete [ 5309.074763] LustreError: 365096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538511 with bad export cookie 2435894019370831529 [ 5309.079123] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5309.084377] LustreError: 365096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5318.697974] Lustre: server umount lustre-MDT0001 complete [ 5322.586077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5324.240398] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5325.399398] LustreError: 369311:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5327.996257] Lustre: 369650:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.115@tcp [ 5328.824367] Lustre: server umount MGS complete [ 5333.260902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.451974] LustreError: 370372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5333.458096] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5333.461722] Lustre: Skipped 1 previous similar message [ 5333.486185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5335.184089] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5338.592954] LustreError: 370373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5339.294633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5339.420812] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5341.099968] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5343.223964] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5344.616897] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5345.331677] Lustre: Permanently reactivating lustre-OST0000 [ 5345.341585] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5345.345879] Lustre: Skipped 2 previous similar messages [ 5367.775160] LustreError: 371833:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5370.724548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5370.877752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5370.880685] Lustre: Skipped 1 previous similar message [ 5373.178687] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5375.437279] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5388.258248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5388.259185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5388.263562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5388.266569] LustreError: Skipped 1 previous similar message [ 5389.969723] Lustre: server umount lustre-OST0000 complete [ 5393.377637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5393.386339] Lustre: Skipped 2 previous similar messages [ 5393.647904] Lustre: server umount lustre-MDT0000 complete [ 5395.452108] LustreError: 370399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538598 with bad export cookie 2435894019370832579 [ 5395.457635] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5395.459926] LustreError: 370399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5395.601199] Lustre: server umount lustre-MDT0001 complete [ 5400.145572] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5401.555596] Key type lgssc unregistered [ 5401.726786] LNet: 374499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5401.731478] LNetError: 374499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5401.741568] LNet: Removed LNI 192.168.203.115@tcp [ 5402.219569] Key type .llcrypt unregistered [ 5402.222744] Key type ._llcrypt unregistered [ 5409.794725] Key type ._llcrypt registered [ 5409.796228] Key type .llcrypt registered [ 5409.846186] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 5417.166480] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5417.546384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5417.553537] alg: No test for adler32 (adler32-zlib) [ 5418.435516] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 5418.539892] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5420.135174] Key type lgssc registered [ 5420.725178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5435.922194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5435.930573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5437.035540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5437.050951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5437.093436] Lustre: lustre-MDT0000: new disk, initializing [ 5437.120572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5437.129425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5438.460413] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5443.549704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.626524] Lustre: 377610:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5443.670047] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5443.674229] Lustre: Skipped 1 previous similar message [ 5443.722308] Lustre: lustre-MDT0001: new disk, initializing [ 5443.769668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5443.791610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5443.799549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5445.394584] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5448.723775] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5450.467630] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5453.753182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5453.910473] Lustre: lustre-OST0000: new disk, initializing [ 5453.914176] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5453.919439] Lustre: 379352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5453.967589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5456.324524] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5458.956655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5458.964055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5459.021894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5459.914899] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5461.427108] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5461.545165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5463.109765] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5463.188399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5469.153930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.154516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5469.158520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5469.160370] LustreError: Skipped 1 previous similar message [ 5469.163955] Lustre: Skipped 1 previous similar message [ 5474.274742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5474.278889] Lustre: Skipped 1 previous similar message [ 5478.367100] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5478.453262] Lustre: server umount lustre-OST0000 complete [ 5484.514182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5484.515572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5484.522448] Lustre: Skipped 2 previous similar messages [ 5484.529859] Lustre: Skipped 1 previous similar message [ 5486.461164] Lustre: server umount lustre-MDT0000 complete [ 5488.191851] LustreError: 377605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538690 with bad export cookie 6586528664624940410 [ 5488.197413] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5488.199815] LustreError: 377605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5488.357800] Lustre: server umount lustre-MDT0001 complete [ 5491.854173] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5492.733657] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:04:55 (1780538695) [ 5495.315697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.539234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5497.153913] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5500.896921] LustreError: 381667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5500.907445] LustreError: 381667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5501.028876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5501.168110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5502.925322] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5505.366570] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5507.005398] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5510.024316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5510.154632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5512.762360] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5515.105539] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5516.054176] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5521.948989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5525.474345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5525.474445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.477110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5525.479611] Lustre: Skipped 1 previous similar message [ 5529.161738] Lustre: server umount lustre-OST0000 complete [ 5530.592471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5530.593132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5530.596537] Lustre: Skipped 1 previous similar message [ 5530.598418] Lustre: Skipped 1 previous similar message [ 5536.430806] Lustre: server umount lustre-MDT0000 complete [ 5538.198417] LustreError: 381646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538740 with bad export cookie 6586528664624941292 [ 5538.202688] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5538.207625] LustreError: 381646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5538.349724] Lustre: server umount lustre-MDT0001 complete [ 5542.738791] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5544.102377] Key type lgssc unregistered [ 5544.267665] LNet: 385036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5544.272682] LNetError: 385036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5544.283642] LNet: Removed LNI 192.168.203.115@tcp [ 5544.688410] Key type .llcrypt unregistered [ 5544.690841] Key type ._llcrypt unregistered [ 5554.013490] Key type ._llcrypt registered [ 5554.015228] Key type .llcrypt registered [ 5554.066799] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5554.228360] Key type .llcrypt unregistered [ 5554.230851] Key type ._llcrypt unregistered [ 5563.221415] Key type ._llcrypt registered [ 5563.222702] Key type .llcrypt registered [ 5563.257347] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5563.651428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5563.664168] alg: No test for adler32 (adler32-zlib) [ 5564.552715] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 5564.676394] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5566.295163] Key type lgssc registered [ 5566.783117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5570.646375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5570.655095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5571.853433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5573.846533] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5578.236727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5578.424175] LustreError: 387194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5578.434756] LustreError: 387194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5578.439742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5581.224859] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5581.538448] LustreError: 387193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5581.550901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5581.551131] LustreError: 387193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5583.951284] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5586.917952] LustreError: 387194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5586.925900] LustreError: 387194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5587.249651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5587.448963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5589.159150] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5597.669067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5597.669512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5597.679105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5602.272356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5602.277854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5602.287171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5603.303572] Lustre: server umount lustre-OST0000 complete [ 5607.906330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.908131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5607.916113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5607.924458] Lustre: Skipped 2 previous similar messages [ 5611.244311] Lustre: server umount lustre-MDT0000 complete [ 5613.026335] LustreError: 387188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5613.037496] LustreError: 387188:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5613.039452] LustreError: 388806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538815 with bad export cookie 8267042030500607897 [ 5613.045673] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5613.050520] LustreError: 388806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5613.188341] Lustre: server umount lustre-MDT0001 complete [ 5618.019833] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5619.665843] Key type lgssc unregistered [ 5619.851611] LNet: 390181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5619.856886] LNetError: 390181:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5619.865746] LNet: Removed LNI 192.168.203.115@tcp [ 5620.369633] Key type .llcrypt unregistered [ 5620.372066] Key type ._llcrypt unregistered [ 5632.396321] Key type ._llcrypt registered [ 5632.399535] Key type .llcrypt registered [ 5632.470245] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5633.004993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5633.021950] alg: No test for adler32 (adler32-zlib) [ 5633.915848] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 5634.013405] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5635.615164] Key type lgssc registered [ 5636.249328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5640.091591] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5640.101972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5655.583325] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5660.703150] LustreError: 391533:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 5687.327215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.688406] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5692.268784] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5695.395052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5695.627116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5695.666824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5697.124558] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5700.650874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5700.826618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5702.409265] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5704.292188] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5705.647092] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5711.329524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5711.330390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.335174] LustreError: Skipped 1 previous similar message [ 5711.344221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5711.352299] Lustre: Skipped 1 previous similar message [ 5712.356297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5712.360985] Lustre: Skipped 1 previous similar message [ 5714.345100] Lustre: server umount lustre-OST0000 complete [ 5716.448382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5716.453365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5716.461092] Lustre: Skipped 1 previous similar message [ 5716.464693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5717.474964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.032396] Lustre: server umount lustre-MDT0000 complete [ 5722.592672] LustreError: 392463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5722.604577] LustreError: 392463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5723.887099] LustreError: 391541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538926 with bad export cookie 11635466824155629591 [ 5723.892566] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5723.894915] LustreError: 391541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5724.043962] Lustre: server umount lustre-MDT0001 complete [ 5728.600461] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5729.885939] Key type lgssc unregistered [ 5730.049515] LNet: 394752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5730.054972] LNetError: 394752:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5730.067516] LNet: Removed LNI 192.168.203.115@tcp [ 5730.428370] Key type .llcrypt unregistered [ 5730.429579] Key type ._llcrypt unregistered [ 5742.593562] Key type ._llcrypt registered [ 5742.595189] Key type .llcrypt registered [ 5742.638935] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5743.077808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5743.086996] alg: No test for adler32 (adler32-zlib) [ 5743.962266] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 5744.058213] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5745.647145] Key type lgssc registered [ 5745.963651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5749.146985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5749.158373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5750.354562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5751.740977] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5755.869408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5756.003178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5757.347892] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5759.915339] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5761.411207] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5764.481905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5764.671994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5767.105085] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5769.035788] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5775.981458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5781.477403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5781.477694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5781.479409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5781.494351] Lustre: Skipped 1 previous similar message [ 5784.552607] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 5784.554770] Lustre: Skipped 2 previous similar messages [ 5786.593681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5786.598758] Lustre: Skipped 2 previous similar messages [ 5786.726972] Lustre: server umount lustre-MDT0000 complete [ 5787.695779] LustreError: 397747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.15@tcp arrived at 1780538990 with bad export cookie 17201123739530191487 [ 5787.702099] LustreError: 397747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5788.516476] LustreError: 396109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780538991 with bad export cookie 17201123739530190829 [ 5788.521726] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5788.523846] LustreError: 396109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5791.713096] LustreError: 396128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5791.713854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5791.714481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5791.714493] Lustre: Skipped 1 previous similar message [ 5791.716043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5791.722795] LustreError: 396128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5793.761146] LustreError: 396839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5793.771595] LustreError: 396839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5794.656894] Lustre: server umount lustre-OST0000 complete [ 5797.868593] Lustre: server umount lustre-MDT0001 complete [ 5802.337108] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5803.469380] Key type lgssc unregistered [ 5803.617697] LNet: 399482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5803.622450] LNetError: 399482:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5803.631523] LNet: Removed LNI 192.168.203.115@tcp [ 5803.939389] Key type .llcrypt unregistered [ 5803.941718] Key type ._llcrypt unregistered [ 5814.149309] Key type ._llcrypt registered [ 5814.151135] Key type .llcrypt registered [ 5814.195291] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5814.582324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5814.588961] alg: No test for adler32 (adler32-zlib) [ 5815.437364] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 5815.519957] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5817.096168] Key type lgssc registered [ 5817.658570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5820.373306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5820.384243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5821.547038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5822.831770] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5826.387841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5826.566352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5827.845112] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5829.700249] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5831.315347] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5834.515182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5834.701806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5837.208709] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5839.683455] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5848.044294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5852.128385] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5852.134781] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5852.145093] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5853.153694] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5853.158295] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.213432] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 5855.218100] Lustre: Skipped 1 previous similar message [ 5857.516587] Lustre: server umount lustre-MDT0001 complete [ 5858.275752] LustreError: 402475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5858.286284] LustreError: 402475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5863.393754] LustreError: 400859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5863.395334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5863.395640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.396862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5865.894969] Lustre: server umount lustre-OST0000 complete [ 5867.801746] Lustre: server umount lustre-MDT0000 complete [ 5873.256641] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5874.744842] Key type lgssc unregistered [ 5874.913651] LNet: 404212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5874.918830] LNetError: 404212:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5874.929553] LNet: Removed LNI 192.168.203.115@tcp [ 5875.375795] Key type .llcrypt unregistered [ 5875.378160] Key type ._llcrypt unregistered [ 5886.289099] Key type ._llcrypt registered [ 5886.291323] Key type .llcrypt registered [ 5886.353175] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 5893.606832] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5893.962259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5894.006201] alg: No test for adler32 (adler32-zlib) [ 5894.891901] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 5895.008094] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5896.623173] Key type lgssc registered [ 5897.285973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5915.478624] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5923.134384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5923.144627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5924.282026] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5924.300406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5924.349936] Lustre: lustre-MDT0000: new disk, initializing [ 5924.383904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.394141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5926.325229] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5934.431727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5934.601579] Lustre: lustre-OST0000: new disk, initializing [ 5934.605213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5934.609144] Lustre: Skipped 1 previous similar message [ 5934.613105] Lustre: 409154:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5934.661047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5937.472983] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5937.760127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5937.766574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5937.787518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5941.577223] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5949.220757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5949.286734] Lustre: 408151:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5949.334042] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5949.385766] Lustre: lustre-MDT0001: new disk, initializing [ 5949.440230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5949.461851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 5949.468516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 5951.208814] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5955.845331] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5955.964754] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5957.745343] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5958.884992] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 5961.201646] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 5963.746380] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5963.747435] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5963.764739] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5963.768786] Lustre: Skipped 1 previous similar message [ 5966.306417] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 5967.337567] Lustre: server umount lustre-MDT0001 complete [ 5973.987435] LustreError: 408160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5973.999283] LustreError: 408160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5979.107544] LustreError: 410865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5979.118136] LustreError: 410865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5983.712435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5983.717565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5983.725511] Lustre: Skipped 1 previous similar message [ 5983.729098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5984.227234] LustreError: 410379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5986.600803] Lustre: server umount lustre-OST0000 complete [ 5994.773243] Lustre: server umount lustre-MDT0000 complete [ 6001.160279] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6002.634301] Key type lgssc unregistered [ 6002.795615] LNet: 412732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6002.800600] LNetError: 412732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6002.814595] LNet: Removed LNI 192.168.203.115@tcp [ 6003.267396] Key type .llcrypt unregistered [ 6003.269741] Key type ._llcrypt unregistered [ 6020.747208] Key type ._llcrypt registered [ 6020.748525] Key type .llcrypt registered [ 6020.787740] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6020.941291] Key type .llcrypt unregistered [ 6020.943633] Key type ._llcrypt unregistered [ 6028.992169] Key type ._llcrypt registered [ 6028.993970] Key type .llcrypt registered [ 6029.050509] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6029.378579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6029.427708] alg: No test for adler32 (adler32-zlib) [ 6030.288026] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 6030.392926] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 6031.983148] Key type lgssc registered [ 6032.549816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6045.154220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6045.165141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6046.284358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6046.297165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6046.335387] Lustre: lustre-MDT0000: new disk, initializing [ 6046.364047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6046.372077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6048.112912] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6053.658746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6053.731130] Lustre: 416050:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6053.773371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6053.777395] Lustre: Skipped 1 previous similar message [ 6053.827317] Lustre: lustre-MDT0001: new disk, initializing [ 6053.869241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6053.885901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6053.891841] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6055.539305] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6059.606861] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6061.407191] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6064.722758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6064.890562] Lustre: lustre-OST0000: new disk, initializing [ 6064.894429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6064.900114] Lustre: 417793:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6064.950094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6066.187689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6066.194602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6066.256189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6067.580219] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6071.595258] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6076.387773] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.389863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6076.396063] Lustre: Skipped 2 previous similar messages [ 6076.400171] Lustre: Skipped 1 previous similar message [ 6080.691506] Lustre: server umount lustre-MDT0000 complete [ 6082.464731] LustreError: 416041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780539285 with bad export cookie 8739068508898590324 [ 6082.470770] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6082.472571] LustreError: 416041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6082.620330] Lustre: server umount lustre-MDT0001 complete [ 6094.648488] Lustre: server umount lustre-OST0000 complete [ 6099.643190] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:15:02 (1780539302) [ 6103.282709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6103.523396] LustreError: 419709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.536259] LustreError: 419709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6103.561946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6105.254228] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6108.640982] LustreError: 419710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6109.057053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6109.258823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6110.781641] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6113.332532] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6115.081342] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6131.300679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6131.419801] Lustre: Found index 0 for lustre-OST0000, updating log [ 6131.523387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6134.001698] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6136.056545] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6149.995511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6155.233756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6155.234943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6155.239044] LustreError: Skipped 1 previous similar message [ 6155.248624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6155.254256] Lustre: Skipped 1 previous similar message [ 6156.321757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6156.325819] Lustre: Skipped 1 previous similar message [ 6159.462213] Lustre: server umount lustre-OST0000 complete [ 6175.712404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6175.717389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6175.725247] Lustre: Skipped 1 previous similar message [ 6175.728841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6177.762267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6178.865279] Lustre: server umount lustre-MDT0000 complete [ 6180.626052] LustreError: 419689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780539383 with bad export cookie 8739068508898591717 [ 6180.631084] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6180.633496] LustreError: 419689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6180.790784] Lustre: server umount lustre-MDT0001 complete [ 6184.638868] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:16:28 (1780539388) [ 6191.199314] LNet: 1 local NIs in recovery (showing 1): 192.168.203.115@tcp [ 6771.853645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6772.094514] LustreError: 425797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6772.137133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6773.740293] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6777.313155] LustreError: 425798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6777.852357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6778.055437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6779.769944] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6782.393731] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6784.146796] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6800.421036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6800.551975] Lustre: Found index 0 for lustre-OST0000, updating log [ 6800.682365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6803.275158] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6805.867389] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6810.084593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6810.086061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6810.093888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6810.101927] Lustre: Skipped 2 previous similar messages [ 6812.964925] Lustre: server umount lustre-OST0000 complete [ 6815.778592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6815.780460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6815.786753] Lustre: Skipped 1 previous similar message [ 6816.801278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6816.805256] Lustre: Skipped 2 previous similar messages [ 6818.849327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6818.853341] Lustre: Skipped 3 previous similar messages [ 6822.945217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6822.949366] Lustre: Skipped 7 previous similar messages [ 6829.023129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6829.089712] LustreError: 425798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6829.097954] LustreError: 425798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6829.105730] Lustre: server umount lustre-MDT0000 complete [ 6830.833713] LustreError: 425823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780540033 with bad export cookie 8739068508898592788 [ 6830.840945] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6830.841426] LustreError: 425823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6830.977341] Lustre: server umount lustre-MDT0001 complete [ 6831.199374] LNet: 1 local NIs in recovery (showing 1): 192.168.203.115@tcp [ 6850.803462] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 6855.474345] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6873.283286] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:27:56 (1780540076) [ 6881.584724] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 6883.158358] Key type lgssc unregistered [ 6883.336971] LNet: 432069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6883.342367] LNetError: 432069:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6931.493696] LNet: Removed LNI 192.168.203.115@tcp [ 6931.968161] Key type .llcrypt unregistered [ 6931.970485] Key type ._llcrypt unregistered [ 6940.748287] Key type ._llcrypt registered [ 6940.750121] Key type .llcrypt registered [ 6940.816204] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6941.306675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6941.325296] alg: No test for adler32 (adler32-zlib) [ 6942.210206] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 6942.328234] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 6943.927153] Key type lgssc registered [ 6944.543257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6946.858142] LNet: Added LNI 192.168.203.115@tcp100 [8/256/0/180] [ 6947.296571] LNet: Added LNI 192.168.203.115@tcp101 [8/256/0/180] [ 6947.742466] LNet: Added LNI 192.168.203.115@tcp102 [8/256/0/180] [ 6948.192753] LNet: Added LNI 192.168.203.115@tcp103 [8/256/0/180] [ 6948.639246] LNet: Added LNI 192.168.203.115@tcp104 [8/256/0/180] [ 6949.073134] LNet: Added LNI 192.168.203.115@tcp105 [8/256/0/180] [ 6949.480824] LNet: Added LNI 192.168.203.115@tcp106 [8/256/0/180] [ 6949.877238] LNet: Added LNI 192.168.203.115@tcp107 [8/256/0/180] [ 6950.282398] LNet: Added LNI 192.168.203.115@tcp108 [8/256/0/180] [ 6950.691798] LNet: Added LNI 192.168.203.115@tcp109 [8/256/0/180] [ 6951.098675] LNet: Added LNI 192.168.203.115@tcp110 [8/256/0/180] [ 6951.529807] LNet: Added LNI 192.168.203.115@tcp111 [8/256/0/180] [ 6951.977113] LNet: Added LNI 192.168.203.115@tcp112 [8/256/0/180] [ 6952.422738] LNet: Added LNI 192.168.203.115@tcp113 [8/256/0/180] [ 6952.862859] LNet: Added LNI 192.168.203.115@tcp114 [8/256/0/180] [ 6953.308210] LNet: Added LNI 192.168.203.115@tcp115 [8/256/0/180] [ 6953.733896] LNet: Added LNI 192.168.203.115@tcp116 [8/256/0/180] [ 6954.166798] LNet: Added LNI 192.168.203.115@tcp117 [8/256/0/180] [ 6954.593586] LNet: Added LNI 192.168.203.115@tcp118 [8/256/0/180] [ 6955.036556] LNet: Added LNI 192.168.203.115@tcp119 [8/256/0/180] [ 6955.461464] LNet: Added LNI 192.168.203.115@tcp120 [8/256/0/180] [ 6955.875170] LNet: Added LNI 192.168.203.115@tcp121 [8/256/0/180] [ 6956.318725] LNet: Added LNI 192.168.203.115@tcp122 [8/256/0/180] [ 6956.768568] LNet: Added LNI 192.168.203.115@tcp123 [8/256/0/180] [ 6957.218482] LNet: Added LNI 192.168.203.115@tcp124 [8/256/0/180] [ 6957.673102] LNet: Added LNI 192.168.203.115@tcp125 [8/256/0/180] [ 6958.115447] LNet: Added LNI 192.168.203.115@tcp126 [8/256/0/180] [ 6958.552118] LNet: Added LNI 192.168.203.115@tcp127 [8/256/0/180] [ 6958.982708] LNet: Added LNI 192.168.203.115@tcp128 [8/256/0/180] [ 6959.406444] LNet: Added LNI 192.168.203.115@tcp129 [8/256/0/180] [ 6959.847461] LNet: Added LNI 192.168.203.115@tcp130 [8/256/0/180] [ 6960.287823] LNet: Added LNI 192.168.203.115@tcp131 [8/256/0/180] [ 6960.736558] LNet: Added LNI 192.168.203.115@tcp132 [8/256/0/180] [ 6961.182270] LNet: Added LNI 192.168.203.115@tcp133 [8/256/0/180] [ 6961.628258] LNet: Added LNI 192.168.203.115@tcp134 [8/256/0/180] [ 6962.072199] LNet: Added LNI 192.168.203.115@tcp135 [8/256/0/180] [ 6965.441166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6965.451579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.548962] Lustre: 435047:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.115@tcp: can't add NID 192.168.203.115@tcp130: rc = -75 [ 6966.598297] LNetError: 432513:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.115 [ 6966.601193] LNetError: Protocol error connecting to 192.168.203.115@tcp119 at host 192.168.203.115:988: is it running a compatible version of Lustre? [ 6966.603996] LNetError: Skipped 1 previous similar message [ 6966.644771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6966.663342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6966.705302] Lustre: lustre-MDT0000: new disk, initializing [ 6966.751234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6966.761626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6967.656922] LNetError: 432512:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.115 [ 6967.663682] LNetError: 432512:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [ 6967.668781] LNetError: Protocol error connecting to 192.168.203.115@tcp107 at host 192.168.203.115:988: is it running a compatible version of Lustre? [ 6968.694333] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6974.364242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6974.455202] Lustre: 435064:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6974.507117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6974.511193] Lustre: Skipped 1 previous similar message [ 6974.567412] Lustre: lustre-MDT0001: new disk, initializing [ 6974.586448] Lustre: 435849:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.115@tcp: can't add NID 192.168.203.115@tcp130: rc = -75 [ 6974.592829] Lustre: 435849:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6974.669956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6974.693058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6974.700761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6976.585827] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6980.714455] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6982.556811] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6985.882765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6986.071253] Lustre: lustre-OST0000: new disk, initializing [ 6986.074497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6986.079209] Lustre: 436814:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6986.133120] Lustre: 436819:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.115@tcp: can't add NID 192.168.203.115@tcp130: rc = -75 [ 6986.136449] Lustre: 436819:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 6986.161641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6988.876923] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6992.903742] Lustre: 437375:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.115@tcp: can't add NID 192.168.203.115@tcp130: rc = -75 [ 6992.909868] Lustre: 437375:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6992.951411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6992.957297] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6993.026838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6993.084894] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6997.984729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6997.990745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6997.999497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6998.501663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6998.505077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6998.510409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7000.736778] LNetError: 435074:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.115 [ 7000.743177] LNetError: Protocol error connecting to 192.168.203.115@tcp119 at host 192.168.203.115:988: is it running a compatible version of Lustre? [ 7002.737440] Lustre: server umount lustre-OST0000 complete [ 7005.664577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7005.669665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7005.678695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7005.682690] Lustre: Skipped 1 previous similar message [ 7006.880726] LNetError: 435070:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.115 [ 7006.887207] LNetError: Protocol error connecting to 192.168.203.115@tcp107 at host 192.168.203.115:988: is it running a compatible version of Lustre? [ 7008.738229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7008.739497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7010.791265] Lustre: server umount lustre-MDT0000 complete [ 7012.560433] LustreError: 435057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780540215 with bad export cookie 7886782064804439054 [ 7012.571280] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7012.717386] Lustre: server umount lustre-MDT0001 complete [ 7013.024606] LNetError: 435069:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.115 [ 7013.031222] LNetError: 435069:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7013.036402] LNetError: Protocol error connecting to 192.168.203.115@tcp119 at host 192.168.203.115:988: is it running a compatible version of Lustre? [ 7013.043121] LNetError: Skipped 1 previous similar message [ 7017.875350] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7019.551356] Key type lgssc unregistered [ 7019.807826] LNet: 438598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7019.815578] LNetError: 438598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7019.820581] LNet: Removed LNI 192.168.203.115@tcp [ 7020.831211] LNet: Removed LNI 192.168.203.115@tcp100 [ 7020.833804] LNetError: 438598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7020.838498] LNetError: 438598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7020.843595] LNet: Removed LNI 192.168.203.115@tcp101 [ 7021.855304] LNet: Removed LNI 192.168.203.115@tcp102 [ 7021.858131] LNetError: 438598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7021.862765] LNetError: 438598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7021.867928] LNet: Removed LNI 192.168.203.115@tcp103 [ 7022.879226] LNet: Removed LNI 192.168.203.115@tcp104 [ 7023.903231] LNet: Removed LNI 192.168.203.115@tcp105 [ 7023.905887] LNetError: 438598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7023.910670] LNetError: 438598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7023.915841] LNet: Removed LNI 192.168.203.115@tcp106 [ 7024.927241] LNet: Removed LNI 192.168.203.115@tcp107 [ 7025.951244] LNet: Removed LNI 192.168.203.115@tcp108 [ 7026.975174] LNet: Removed LNI 192.168.203.115@tcp109 [ 7027.999247] LNet: Removed LNI 192.168.203.115@tcp110 [ 7028.002967] LNetError: 438598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7028.009612] LNetError: 438598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7028.016857] LNet: Removed LNI 192.168.203.115@tcp111 [ 7029.023160] LNet: Removed LNI 192.168.203.115@tcp112 [ 7030.047172] LNet: Removed LNI 192.168.203.115@tcp113 [ 7031.071183] LNet: Removed LNI 192.168.203.115@tcp114 [ 7032.095173] LNet: Removed LNI 192.168.203.115@tcp115 [ 7033.119175] LNet: Removed LNI 192.168.203.115@tcp116 [ 7034.143239] LNet: Removed LNI 192.168.203.115@tcp117 [ 7035.167172] LNet: Removed LNI 192.168.203.115@tcp118 [ 7036.191207] LNet: Removed LNI 192.168.203.115@tcp119 [ 7036.194958] LNetError: 438598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7036.201573] LNetError: 438598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7036.208927] LNet: Removed LNI 192.168.203.115@tcp120 [ 7036.212550] LNet: Removed LNI 192.168.203.115@tcp121 [ 7036.216201] LNet: Removed LNI 192.168.203.115@tcp122 [ 7036.219820] LNet: Removed LNI 192.168.203.115@tcp123 [ 7036.223446] LNet: Removed LNI 192.168.203.115@tcp124 [ 7036.227105] LNet: Removed LNI 192.168.203.115@tcp125 [ 7036.230675] LNet: Removed LNI 192.168.203.115@tcp126 [ 7036.234380] LNet: Removed LNI 192.168.203.115@tcp127 [ 7036.237937] LNet: Removed LNI 192.168.203.115@tcp128 [ 7036.241560] LNet: Removed LNI 192.168.203.115@tcp129 [ 7036.245202] LNet: Removed LNI 192.168.203.115@tcp130 [ 7036.248804] LNet: Removed LNI 192.168.203.115@tcp131 [ 7036.252494] LNet: Removed LNI 192.168.203.115@tcp132 [ 7036.256151] LNet: Removed LNI 192.168.203.115@tcp133 [ 7036.259709] LNet: Removed LNI 192.168.203.115@tcp134 [ 7036.270406] LNet: Removed LNI 192.168.203.115@tcp135 [ 7036.723672] Key type .llcrypt unregistered [ 7036.726651] Key type ._llcrypt unregistered [ 7047.248156] Key type ._llcrypt registered [ 7047.249954] Key type .llcrypt registered [ 7047.316870] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 7055.451099] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7055.848182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7055.918112] alg: No test for adler32 (adler32-zlib) [ 7056.849463] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 7056.983264] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7058.599168] Key type lgssc registered [ 7059.245407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7076.362810] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:31:19 (1780540279) [ 7084.843446] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7086.433335] Key type lgssc unregistered [ 7086.605731] LNet: 442621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7086.610636] LNetError: 442621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7086.621682] LNet: Removed LNI 192.168.203.115@tcp [ 7087.064633] Key type .llcrypt unregistered [ 7087.067042] Key type ._llcrypt unregistered [ 7095.576182] Key type ._llcrypt registered [ 7095.578022] Key type .llcrypt registered [ 7095.644788] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7096.086187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7096.119149] alg: No test for adler32 (adler32-zlib) [ 7097.018698] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 7097.144689] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7098.751148] Key type lgssc registered [ 7099.377634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7104.107513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7104.118173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7105.255392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7105.275930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7105.325590] Lustre: lustre-MDT0000: new disk, initializing [ 7105.360498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7105.370817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7107.166300] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7112.799159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7112.899592] Lustre: 443794:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7112.948058] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7112.952052] Lustre: Skipped 1 previous similar message [ 7113.002885] Lustre: lustre-MDT0001: new disk, initializing [ 7113.058451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7113.080523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7113.088286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7115.007715] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7119.116645] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7120.867553] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7124.190045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7124.339181] Lustre: lustre-OST0000: new disk, initializing [ 7124.342419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7124.347203] Lustre: 445538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7124.397395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7127.151824] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7130.630430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7130.636341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7130.685116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7131.069604] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7132.758688] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing lnet_if_list [ 7135.157940] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7135.178277] LNet: Added LNI 192.168.203.115@tcp10 [8/256/0/180] [ 7137.871296] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7137.890831] LNet: Added LNI 192.168.203.115@tcp11 [8/256/0/180] [ 7139.791082] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7139.811811] LNet: Added LNI 192.168.203.115@tcp12 [8/256/0/180] [ 7141.796559] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7141.816617] LNet: Added LNI 192.168.203.115@tcp13 [8/256/0/180] [ 7143.858946] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7143.878676] LNet: Added LNI 192.168.203.115@tcp14 [8/256/0/180] [ 7145.916325] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7145.936863] LNet: Added LNI 192.168.203.115@tcp15 [8/256/0/180] [ 7148.061630] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7148.081349] LNet: Added LNI 192.168.203.115@tcp16 [8/256/0/180] [ 7150.331939] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7150.351957] LNet: Added LNI 192.168.203.115@tcp17 [8/256/0/180] [ 7152.687575] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7152.705782] LNet: Added LNI 192.168.203.115@tcp18 [8/256/0/180] [ 7154.825248] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7154.844705] LNet: Added LNI 192.168.203.115@tcp19 [8/256/0/180] [ 7157.209787] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7157.229249] LNet: Added LNI 192.168.203.115@tcp20 [8/256/0/180] [ 7159.644911] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7159.665384] LNet: Added LNI 192.168.203.115@tcp21 [8/256/0/180] [ 7162.111079] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7162.130289] LNet: Added LNI 192.168.203.115@tcp22 [8/256/0/180] [ 7164.662983] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7164.684034] LNet: Added LNI 192.168.203.115@tcp23 [8/256/0/180] [ 7167.239460] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7167.259285] LNet: Added LNI 192.168.203.115@tcp24 [8/256/0/180] [ 7169.940995] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7169.964588] LNet: Added LNI 192.168.203.115@tcp25 [8/256/0/180] [ 7172.852467] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7172.876176] LNet: Added LNI 192.168.203.115@tcp26 [8/256/0/180] [ 7175.583308] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7175.607458] LNet: Added LNI 192.168.203.115@tcp27 [8/256/0/180] [ 7178.368289] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7178.388551] LNet: Added LNI 192.168.203.115@tcp28 [8/256/0/180] [ 7181.340984] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7181.362665] LNet: Added LNI 192.168.203.115@tcp29 [8/256/0/180] [ 7184.365300] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7184.382990] LNet: Added LNI 192.168.203.115@tcp30 [8/256/0/180] [ 7187.405404] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7187.420996] LNet: Added LNI 192.168.203.115@tcp31 [8/256/0/180] [ 7190.414520] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7190.430826] LNet: Added LNI 192.168.203.115@tcp32 [8/256/0/180] [ 7193.407716] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7193.428777] LNet: Added LNI 192.168.203.115@tcp33 [8/256/0/180] [ 7196.803470] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7196.824498] LNet: Added LNI 192.168.203.115@tcp34 [8/256/0/180] [ 7200.085723] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7200.110549] LNet: Added LNI 192.168.203.115@tcp35 [8/256/0/180] [ 7203.376599] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7203.396558] LNet: Added LNI 192.168.203.115@tcp36 [8/256/0/180] [ 7206.594647] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7206.612370] LNet: Added LNI 192.168.203.115@tcp37 [8/256/0/180] [ 7210.000344] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7210.020571] LNet: Added LNI 192.168.203.115@tcp38 [8/256/0/180] [ 7213.378481] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7213.399154] LNet: Added LNI 192.168.203.115@tcp39 [8/256/0/180] [ 7216.886516] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7216.906573] LNet: Added LNI 192.168.203.115@tcp40 [8/256/0/180] [ 7220.395603] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7220.416321] LNet: Added LNI 192.168.203.115@tcp41 [8/256/0/180] [ 7224.173979] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7224.194349] LNet: Added LNI 192.168.203.115@tcp42 [8/256/0/180] [ 7227.859252] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7227.892620] LNet: Added LNI 192.168.203.115@tcp43 [8/256/0/180] [ 7231.625139] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7231.645568] LNet: Added LNI 192.168.203.115@tcp44 [8/256/0/180] [ 7235.467385] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7235.487580] LNet: Added LNI 192.168.203.115@tcp45 [8/256/0/180] [ 7239.364615] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7239.387950] LNet: Added LNI 192.168.203.115@tcp46 [8/256/0/180] [ 7243.332452] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7243.353826] LNet: Added LNI 192.168.203.115@tcp47 [8/256/0/180] [ 7247.458989] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7247.479341] LNet: Added LNI 192.168.203.115@tcp48 [8/256/0/180] [ 7251.333762] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7251.353754] LNet: Added LNI 192.168.203.115@tcp49 [8/256/0/180] [ 7255.454539] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7255.474880] LNet: Added LNI 192.168.203.115@tcp50 [8/256/0/180] [ 7266.784559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7266.789764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7266.798775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7268.836345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7268.837183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7268.840397] Lustre: Skipped 1 previous similar message [ 7268.848391] Lustre: Skipped 1 previous similar message [ 7271.670533] Lustre: server umount lustre-MDT0000 complete [ 7273.478298] LustreError: 443786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780540476 with bad export cookie 12872730055845800836 [ 7273.489298] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7273.652277] Lustre: server umount lustre-MDT0001 complete [ 7282.953270] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7287.093113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7289.311264] Lustre: 443070:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780540476/real 1780540476] req@00000000d337cb1b x1867031824663040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780540492 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7289.327331] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.431164] Lustre: 443067:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780540481/real 1780540481] req@0000000003dd60b5 x1867031824663168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780540497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7298.592553] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb2a51fc87c2ef3bf [ 7298.598184] Lustre: MGC192.168.203.115@tcp: Connection restored to 0@lo (at 0@lo) [ 7298.798597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7299.551283] Lustre: 443068:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780540486/real 1780540486] req@00000000561b7514 x1867031824663424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780540502 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7300.719770] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7304.927252] Lustre: 443069:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780540491/real 1780540491] req@000000003f00faa7 x1867031824663680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780540507 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7304.931482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7305.154948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7307.065884] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7308.264164] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7309.655211] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7311.310201] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7323.620775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7323.628484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7323.628974] Lustre: Skipped 1 previous similar message [ 7327.149152] Lustre: server umount lustre-OST0000 complete [ 7330.784539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7330.789731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7330.798534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7330.802644] Lustre: Skipped 1 previous similar message [ 7335.153652] Lustre: server umount lustre-MDT0000 complete [ 7336.943334] LustreError: 443787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780540539 with bad export cookie 12872730055845802943 [ 7336.950983] LustreError: 443787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7336.952825] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7337.123906] Lustre: server umount lustre-MDT0001 complete [ 7341.876734] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7343.343684] Key type lgssc unregistered [ 7343.558413] LNet: 471467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7343.564731] LNetError: 471467:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7343.569562] LNet: Removed LNI 192.168.203.115@tcp [ 7343.572104] LNet: Removed LNI 192.168.203.115@tcp10 [ 7343.574391] LNet: Removed LNI 192.168.203.115@tcp11 [ 7343.576644] LNet: Removed LNI 192.168.203.115@tcp12 [ 7343.579176] LNet: Removed LNI 192.168.203.115@tcp13 [ 7343.581483] LNet: Removed LNI 192.168.203.115@tcp14 [ 7343.583619] LNet: Removed LNI 192.168.203.115@tcp15 [ 7343.585759] LNet: Removed LNI 192.168.203.115@tcp16 [ 7343.587694] LNet: Removed LNI 192.168.203.115@tcp17 [ 7343.589648] LNet: Removed LNI 192.168.203.115@tcp18 [ 7343.591755] LNet: Removed LNI 192.168.203.115@tcp19 [ 7343.593747] LNet: Removed LNI 192.168.203.115@tcp20 [ 7343.595649] LNet: Removed LNI 192.168.203.115@tcp21 [ 7343.597722] LNet: Removed LNI 192.168.203.115@tcp22 [ 7343.600789] LNet: Removed LNI 192.168.203.115@tcp23 [ 7343.602784] LNet: Removed LNI 192.168.203.115@tcp24 [ 7343.604713] LNet: Removed LNI 192.168.203.115@tcp25 [ 7343.606509] LNet: Removed LNI 192.168.203.115@tcp26 [ 7343.608411] LNet: Removed LNI 192.168.203.115@tcp27 [ 7343.610288] LNet: Removed LNI 192.168.203.115@tcp28 [ 7343.612232] LNet: Removed LNI 192.168.203.115@tcp29 [ 7343.614183] LNet: Removed LNI 192.168.203.115@tcp30 [ 7343.615962] LNet: Removed LNI 192.168.203.115@tcp31 [ 7343.617605] LNet: Removed LNI 192.168.203.115@tcp32 [ 7343.619225] LNet: Removed LNI 192.168.203.115@tcp33 [ 7343.620735] LNet: Removed LNI 192.168.203.115@tcp34 [ 7343.622209] LNet: Removed LNI 192.168.203.115@tcp35 [ 7343.623747] LNet: Removed LNI 192.168.203.115@tcp36 [ 7343.625225] LNet: Removed LNI 192.168.203.115@tcp37 [ 7343.626618] LNet: Removed LNI 192.168.203.115@tcp38 [ 7343.628090] LNet: Removed LNI 192.168.203.115@tcp39 [ 7343.629541] LNet: Removed LNI 192.168.203.115@tcp40 [ 7343.631014] LNet: Removed LNI 192.168.203.115@tcp41 [ 7343.632323] LNet: Removed LNI 192.168.203.115@tcp42 [ 7343.633727] LNet: Removed LNI 192.168.203.115@tcp43 [ 7343.635119] LNet: Removed LNI 192.168.203.115@tcp44 [ 7343.636470] LNet: Removed LNI 192.168.203.115@tcp45 [ 7343.637840] LNet: Removed LNI 192.168.203.115@tcp46 [ 7343.639199] LNet: Removed LNI 192.168.203.115@tcp47 [ 7343.640542] LNet: Removed LNI 192.168.203.115@tcp48 [ 7343.641915] LNet: Removed LNI 192.168.203.115@tcp49 [ 7343.650595] LNet: Removed LNI 192.168.203.115@tcp50 [ 7344.043453] Key type .llcrypt unregistered [ 7344.045764] Key type ._llcrypt unregistered [ 7353.251233] Key type ._llcrypt registered [ 7353.253108] Key type .llcrypt registered [ 7353.316489] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 7361.474292] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7361.864280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7361.902075] alg: No test for adler32 (adler32-zlib) [ 7362.773283] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 7362.897076] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7364.511169] Key type lgssc registered [ 7365.091361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7382.197520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7382.207773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.321832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7383.338270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7383.385332] Lustre: lustre-MDT0000: new disk, initializing [ 7383.414416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7383.424775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7385.105562] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7390.598812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.663612] Lustre: 474605:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7390.706851] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7390.710973] Lustre: Skipped 1 previous similar message [ 7390.760609] Lustre: lustre-MDT0001: new disk, initializing [ 7390.796086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7390.810633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7390.813844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7392.522048] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7396.658790] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7398.378598] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7402.099756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7402.223368] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7402.277426] Lustre: lustre-OST0000: new disk, initializing [ 7402.280938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7402.286319] Lustre: 476398:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7402.336452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7404.864118] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7408.932338] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7411.210099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7411.218368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7411.279266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7416.293676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7416.294638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7416.298702] LustreError: Skipped 1 previous similar message [ 7416.307766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7416.313894] Lustre: Skipped 1 previous similar message [ 7418.603195] Lustre: server umount lustre-OST0000 complete [ 7421.410389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7421.410822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7421.423616] Lustre: Skipped 1 previous similar message [ 7421.427796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7421.431930] Lustre: Skipped 1 previous similar message [ 7426.548617] Lustre: server umount lustre-MDT0000 complete [ 7428.313167] LustreError: 474598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780540631 with bad export cookie 4754633775279339967 [ 7428.320560] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7428.320714] LustreError: 474598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7428.482158] Lustre: server umount lustre-MDT0001 complete [ 7433.062278] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7434.501023] Key type lgssc unregistered [ 7434.685664] LNet: 478171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7434.690901] LNetError: 478171:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7434.705577] LNet: Removed LNI 192.168.203.115@tcp [ 7435.172448] Key type .llcrypt unregistered [ 7435.174780] Key type ._llcrypt unregistered [ 7444.701909] Key type ._llcrypt registered [ 7444.703199] Key type .llcrypt registered [ 7444.745098] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7445.090092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7445.111057] alg: No test for adler32 (adler32-zlib) [ 7446.020764] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 7446.133284] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7447.743164] Key type lgssc registered [ 7448.377793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7452.330650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7452.342660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7453.581747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7455.446820] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7459.614880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.778661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7461.531108] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7464.141712] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7465.997238] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7469.232618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7469.400860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7471.993585] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7474.620409] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7484.958099] loop0: detected capacity change from 0 to 204800000 [ 7485.100556] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7485.108374] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7485.116361] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7488.084561] loop0: detected capacity change from 0 to 204800000 [ 7488.155429] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7488.159849] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7488.164181] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7490.607920] loop0: detected capacity change from 0 to 204800000 [ 7493.143689] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7493.230449] Lustre: fs15246-OST0000: new disk, initializing [ 7493.234069] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7493.240117] Lustre: 482661:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7493.272940] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7495.888709] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7501.043078] loop1: detected capacity change from 0 to 204800000 [ 7503.643560] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7503.699887] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7503.738891] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7503.794033] Lustre: fs15246-MDT0000: new disk, initializing [ 7503.860301] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7503.873828] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7503.939168] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7504.992603] Lustre: 483764:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000000fb3f150 x1867032190550272/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780540723 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7505.700404] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7514.083755] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7514.089811] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7514.105120] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7520.744223] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.753357] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7521.966372] Lustre: server umount fs15246-MDT0000 complete [ 7525.862267] LustreError: 479400:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7525.873207] LustreError: 479400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7526.286900] Lustre: server umount fs15246-OST0000 complete [ 7530.979214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7530.980508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7530.987514] Lustre: Skipped 1 previous similar message [ 7530.994161] Lustre: Skipped 1 previous similar message [ 7536.101680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7536.105810] Lustre: Skipped 1 previous similar message [ 7541.218846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7541.222876] Lustre: Skipped 1 previous similar message [ 7545.311178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7545.389916] Lustre: server umount lustre-OST0000 complete [ 7551.457545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7551.459155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7551.465223] Lustre: Skipped 1 previous similar message [ 7553.323864] Lustre: server umount lustre-MDT0000 complete [ 7555.088040] LustreError: 479384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780540757 with bad export cookie 6764191561371833037 [ 7555.093368] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7555.095789] LustreError: 479384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7555.246354] Lustre: server umount lustre-MDT0001 complete [ 7559.909846] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7561.357983] Key type lgssc unregistered [ 7561.527694] LNet: 486827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7561.532836] LNetError: 486827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7561.544610] LNet: Removed LNI 192.168.203.115@tcp [ 7561.981110] Key type .llcrypt unregistered [ 7561.983354] Key type ._llcrypt unregistered [ 7571.625550] Key type ._llcrypt registered [ 7571.627420] Key type .llcrypt registered [ 7571.690049] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7572.008548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7572.015520] alg: No test for adler32 (adler32-zlib) [ 7572.902072] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 7572.995288] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7574.591166] Key type lgssc registered [ 7575.212491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7593.792550] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 7598.373133] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7615.771900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7615.783642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7616.901261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7616.914713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7616.955676] Lustre: lustre-MDT0000: new disk, initializing [ 7616.987897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7616.999243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7618.792333] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7624.429990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7624.502349] Lustre: 491218:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7624.573166] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7624.575796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7624.579388] Lustre: Skipped 1 previous similar message [ 7624.623316] Lustre: lustre-MDT0001: new disk, initializing [ 7624.666684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7624.686203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7624.690275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7626.487734] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7630.625979] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7632.506461] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7635.861410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7636.022154] Lustre: lustre-OST0000: new disk, initializing [ 7636.025578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7636.030560] Lustre: 492961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7636.086430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7638.677393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7638.683425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7638.703963] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7638.741504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7642.822245] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7644.832378] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7644.975314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7646.937626] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7647.072611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7649.253335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7649.255221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7649.258419] LustreError: Skipped 1 previous similar message [ 7649.267898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7649.273223] Lustre: Skipped 1 previous similar message [ 7654.130957] Lustre: server umount lustre-OST0000 complete [ 7659.490818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7659.492438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7659.499074] Lustre: Skipped 2 previous similar messages [ 7659.505883] Lustre: Skipped 1 previous similar message [ 7664.609589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7664.613621] Lustre: Skipped 1 previous similar message [ 7669.729662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7669.733747] Lustre: Skipped 1 previous similar message [ 7670.239113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7670.337767] Lustre: server umount lustre-MDT0000 complete [ 7671.951267] LustreError: 491210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780540874 with bad export cookie 11504982298746369975 [ 7671.957126] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7672.144794] Lustre: server umount lustre-MDT0001 complete [ 7684.050726] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 7689.151514] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7704.979823] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7538 sec ========== 22:41:48 (1780540908) [ 7705.868398] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:41:49 (1780540909) === [ 7707.565562] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:41:50 (1780540910) === [ 7726.257599] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7727.750346] Key type lgssc unregistered [ 7727.920663] LNet: 499400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7727.925812] LNetError: 499400:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7727.935497] LNet: Removed LNI 192.168.203.115@tcp [ 7728.378566] Key type .llcrypt unregistered [ 7728.380889] Key type ._llcrypt unregistered