[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 411423662 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001007] APIC: Switch to symmetric I/O mode setup [ 0.002153] x2apic enabled [ 0.002928] Switched APIC routing to physical x2apic. [ 0.003006] kvm-guest: setup PV IPIs [ 0.005850] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006007] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007004] pid_max: default: 32768 minimum: 301 [ 0.008062] LSM: Security Framework initializing [ 0.009023] Yama: becoming mindful. [ 0.010016] SELinux: Initializing. [ 0.011038] *** VALIDATE selinux *** [ 0.017614] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021298] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022091] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023060] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024050] *** VALIDATE tmpfs *** [ 0.025320] *** VALIDATE proc *** [ 0.026137] *** VALIDATE cgroup *** [ 0.027004] *** VALIDATE cgroup2 *** [ 0.028171] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.029095] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.031015] Spectre V2 : User space: Vulnerable [ 0.032003] Speculative Store Bypass: Vulnerable [ 0.035195] debug: unmapping init [mem 0xffffffff96659000-0xffffffff96660fff] [ 0.037137] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.038408] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.039010] ... version: 2 [ 0.039821] ... bit width: 48 [ 0.040006] ... generic registers: 4 [ 0.040699] ... value mask: 0000ffffffffffff [ 0.041005] ... max period: 00007fffffffffff [ 0.042005] ... fixed-purpose events: 3 [ 0.042792] ... event mask: 000000070000000f [ 0.043105] rcu: Hierarchical SRCU implementation. [ 0.044980] smp: Bringing up secondary CPUs ... [ 0.045344] x86: Booting SMP configuration: [ 0.046010] .... node #0, CPUs: #1 #2 #3 [ 0.050014] smp: Brought up 1 node, 4 CPUs [ 0.051007] smpboot: Max logical packages: 1 [ 0.051857] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.257584] node 0 deferred pages initialised in 205ms [ 0.260611] devtmpfs: initialized [ 0.261179] x86/mm: Memory block size: 128MB [ 0.263223] gcov: version magic: 0x41383552 [ 0.264612] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.267114] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.269371] pinctrl core: initialized pinctrl subsystem [ 0.272119] [ 0.272468] ************************************************************* [ 0.274010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.276006] ** ** [ 0.278006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.281008] ** ** [ 0.283008] ** This means that this kernel is built to expose internal ** [ 0.285008] ** IOMMU data structures, which may compromise security on ** [ 0.287009] ** your system. ** [ 0.289008] ** ** [ 0.291008] ** If you see this message and you are not debugging the ** [ 0.293008] ** kernel, report this immediately to your vendor! ** [ 0.294006] ** ** [ 0.296006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.298007] ************************************************************* [ 0.300572] NET: Registered protocol family 16 [ 0.301341] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.304031] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.306030] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.308061] cpuidle: using governor menu [ 0.309398] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.311254] PCI: Using configuration type 1 for base access [ 0.312091] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.321100] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.324016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.327085] cryptd: max_cpu_qlen set to 1000 [ 0.330217] ACPI: Added _OSI(Module Device) [ 0.332019] ACPI: Added _OSI(Processor Device) [ 0.334020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.336013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.341231] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.346521] ACPI: Interpreter enabled [ 0.348043] ACPI: PM: (supports S0 S3 S4 S5) [ 0.350012] ACPI: Using IOAPIC for interrupt routing [ 0.352089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.355371] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.365000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.367022] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.369010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.372042] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.377073] acpiphp: Slot [2] registered [ 0.378072] acpiphp: Slot [5] registered [ 0.379093] acpiphp: Slot [6] registered [ 0.381077] acpiphp: Slot [7] registered [ 0.382075] acpiphp: Slot [8] registered [ 0.383059] acpiphp: Slot [9] registered [ 0.384064] acpiphp: Slot [10] registered [ 0.385065] acpiphp: Slot [3] registered [ 0.386052] acpiphp: Slot [4] registered [ 0.387064] acpiphp: Slot [11] registered [ 0.388051] acpiphp: Slot [12] registered [ 0.389053] acpiphp: Slot [13] registered [ 0.389989] acpiphp: Slot [14] registered [ 0.391055] acpiphp: Slot [15] registered [ 0.392048] acpiphp: Slot [16] registered [ 0.393065] acpiphp: Slot [17] registered [ 0.394073] acpiphp: Slot [18] registered [ 0.395052] acpiphp: Slot [19] registered [ 0.397051] acpiphp: Slot [20] registered [ 0.398050] acpiphp: Slot [21] registered [ 0.399077] acpiphp: Slot [22] registered [ 0.400048] acpiphp: Slot [23] registered [ 0.401057] acpiphp: Slot [24] registered [ 0.402051] acpiphp: Slot [25] registered [ 0.403050] acpiphp: Slot [26] registered [ 0.404051] acpiphp: Slot [27] registered [ 0.405050] acpiphp: Slot [28] registered [ 0.406050] acpiphp: Slot [29] registered [ 0.407072] acpiphp: Slot [30] registered [ 0.408050] acpiphp: Slot [31] registered [ 0.409040] PCI host bridge to bus 0000:00 [ 0.410010] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.412009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.414010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.416009] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.418009] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.419015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.421135] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.423750] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.426019] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.433008] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.437042] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.439009] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.441008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.443008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.444340] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.446571] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.448021] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.449469] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.453008] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.460979] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.464007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.466818] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.469988] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.474008] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.484009] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.489000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.495008] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.502008] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.519022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.527089] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.534010] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.537894] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.550013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.559879] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.567011] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.577013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.593013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.604739] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.612015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.618013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.634016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.646067] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.652014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.660014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.677015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.690264] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.693400] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.697532] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.700365] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.703226] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.709004] iommu: Default domain type: Passthrough [ 0.711511] SCSI subsystem initialized [ 0.713164] ACPI: bus type USB registered [ 0.715113] usbcore: registered new interface driver usbfs [ 0.718067] usbcore: registered new interface driver hub [ 0.720063] usbcore: registered new device driver usb [ 0.723178] pps_core: LinuxPPS API ver. 1 registered [ 0.725012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.729032] PTP clock support registered [ 0.732122] EDAC MC: Ver: 3.0.0 [ 0.734183] PCI: Using ACPI for IRQ routing [ 0.737165] NetLabel: Initializing [ 0.739013] NetLabel: domain hash size = 128 [ 0.741012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.744072] NetLabel: unlabeled traffic allowed by default [ 0.747116] vgaarb: loaded [ 0.749170] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.751017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.757434] clocksource: Switched to clocksource kvm-clock [ 0.855692] VFS: Disk quotas dquot_6.6.0 [ 0.856639] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.857993] *** VALIDATE ramfs *** [ 0.858734] *** VALIDATE hugetlbfs *** [ 0.859515] pnp: PnP ACPI init [ 0.860979] pnp: PnP ACPI: found 6 devices [ 0.873891] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.875990] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.877307] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.878791] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.880292] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.881897] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.883712] NET: Registered protocol family 2 [ 0.885209] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.888525] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.890696] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.894280] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.898426] TCP: Hash tables configured (established 65536 bind 65536) [ 0.902224] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.906417] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.909912] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.912954] NET: Registered protocol family 1 [ 0.915439] RPC: Registered named UNIX socket transport module. [ 0.916712] RPC: Registered udp transport module. [ 0.917645] RPC: Registered tcp transport module. [ 0.918603] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.919905] NET: Registered protocol family 44 [ 0.920804] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.921993] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.923142] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.924475] PCI: CLS 0 bytes, default 64 [ 0.925410] Unpacking initramfs... [ 2.089121] debug: unmapping init [mem 0xffff9c9b7cc54000-0xffff9c9b7ffbffff] [ 2.094257] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.095792] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.098992] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.547259] Initialise system trusted keyrings [ 2.548414] Key type blacklist registered [ 2.549729] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.560219] zbud: loaded [ 2.563210] *** VALIDATE nfs *** [ 2.564028] *** VALIDATE nfs4 *** [ 2.565044] pstore: using deflate compression [ 2.566803] Platform Keyring initialized [ 2.648427] NET: Registered protocol family 38 [ 2.649899] Key type asymmetric registered [ 2.650793] Asymmetric key parser 'x509' registered [ 2.651784] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.653623] io scheduler mq-deadline registered [ 2.654778] io scheduler kyber registered [ 2.655940] io scheduler bfq registered [ 2.657596] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.659979] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.662308] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.664552] ACPI: Power Button [PWRF] [ 2.668821] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.673729] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.682370] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.687270] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.698086] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.723968] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.749012] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.752401] Non-volatile memory driver v1.3 [ 2.753318] Linux agpgart interface v0.103 [ 2.788805] virtio_blk virtio1: [vda] 145160 512-byte logical blocks (74.3 MB/70.9 MiB) [ 2.792945] vda: detected capacity change from 0 to 74321920 [ 2.810421] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.814511] vdb: detected capacity change from 0 to 1073741824 [ 2.832365] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.836573] vdc: detected capacity change from 0 to 2621440000 [ 2.854280] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.858403] vdd: detected capacity change from 0 to 2621440000 [ 2.875976] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.879759] vde: detected capacity change from 0 to 4294967296 [ 2.895741] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.899017] vdf: detected capacity change from 0 to 4294967296 [ 2.904740] libphy: Fixed MDIO Bus: probed [ 2.910601] usbcore: registered new interface driver usbserial_generic [ 2.914047] usbserial: USB Serial support registered for generic [ 2.917384] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.923227] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.925697] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.928653] mousedev: PS/2 mouse device common for all mice [ 2.932086] rtc_cmos 00:05: RTC can wake from S4 [ 2.935924] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.936269] rtc_cmos 00:05: registered as rtc0 [ 2.943353] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.943933] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.945193] intel_pstate: CPU model not supported [ 2.950788] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.955830] hid: raw HID events driver (C) Jiri Kosina [ 2.957101] usbcore: registered new interface driver usbhid [ 2.958189] usbhid: USB HID core driver [ 2.959095] drop_monitor: Initializing network drop monitor service [ 2.960573] Initializing XFRM netlink socket [ 2.961944] NET: Registered protocol family 10 [ 2.964482] Segment Routing with IPv6 [ 2.965386] NET: Registered protocol family 17 [ 2.967282] mpls_gso: MPLS GSO support [ 2.976123] RAS: Correctable Errors collector initialized. [ 2.979028] AVX version of gcm_enc/dec engaged. [ 2.981277] AES CTR mode by8 optimization enabled [ 3.060441] sched_clock: Marking stable (3060405579, 0)->(3652714098, -592308519) [ 3.064652] registered taskstats version 1 [ 3.067403] Loading compiled-in X.509 certificates [ 3.069887] zswap: loaded using pool lzo/zbud [ 3.093003] Key type big_key registered [ 3.103643] Key type encrypted registered [ 3.105200] ima: No TPM chip found, activating TPM-bypass! [ 3.107251] ima: Allocated hash algorithm: sha1 [ 3.109021] ima: No architecture policies found [ 3.110612] evm: Initialising EVM extended attributes: [ 3.112336] evm: security.selinux [ 3.113627] evm: security.ima [ 3.114696] evm: security.capability [ 3.115915] evm: HMAC attrs: 0x1 [ 3.117878] rtc_cmos 00:05: setting system clock to 2026-06-30 22:07:45 UTC (1782857265) [ 3.123558] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 3.126177] debug: unmapping init [mem 0xffffffff96382000-0xffffffff96658fff] [ 3.132156] Write protecting the kernel read-only data: 28672k [ 3.134810] debug: unmapping init [mem 0xffffffff94a03000-0xffffffff94bfffff] [ 3.137042] debug: unmapping init [mem 0xffffffff95314000-0xffffffff953fffff] [ 3.160354] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.166337] systemd[1]: Detected virtualization kvm. [ 3.167716] systemd[1]: Detected architecture x86-64. [ 3.169039] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.189509] systemd[1]: No hostname configured. [ 3.190690] systemd[1]: Set hostname to . [ 3.191967] random: systemd: uninitialized urandom read (16 bytes read) [ 3.193599] systemd[1]: Initializing machine ID from random generator. [ 3.293925] random: systemd: uninitialized urandom read (16 bytes read) [ 3.295837] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.298696] random: systemd: uninitialized urandom read (16 bytes read) [ 3.300457] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.305082] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.766710] device-mapper: uevent: version 1.0.3 [ 3.768363] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices.[ 4.356104] random: fast init done [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.429157] virtio_net virtio0 ens2: renamed from eth0 [ 4.469456] scsi host0: ata_piix [ 4.492473] scsi host1: ata_piix [ 4.493436] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.494975] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.450331] dracut-initqueue[588]: RTNETLINK answers: File exists [ 8.621259] random: crng init done [ 8.621899] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 8.960427] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 9.932715] printk: systemd: 25 output lines suppressed due to ratelimiting [ 10.077943] SELinux: Disabled at runtime. [ 10.129871] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.134391] systemd[1]: Detected virtualization kvm. [ 10.135455] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 10.438075] systemd[1]: initrd-switch-root.service: Succeeded. [ 10.440024] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 10.442413] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 10.444411] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 10.446193] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 10.451234] systemd[1]: Starting Journal Service... Starting Journal Service... [ 10.457154] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ 10.549719] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 10.768310] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 11.067419] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 11.076702] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 11.215988] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 11.222862] EDAC sbridge: Ver: 1.1.2 [ 12.338860] Key type dns_resolver registered [ 12.623670] NFS: Registering the id_resolver key type [ 12.624893] Key type id_resolver registered [ 12.625868] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg413-server login: [ 38.926798] libcfs: loading out-of-tree module taints kernel. [ 38.946449] Key type ._llcrypt registered [ 38.947514] Key type .llcrypt registered [ 38.989194] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 48.411617] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 48.993301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 48.999088] alg: No test for adler32 (adler32-zlib) [ 50.058593] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 50.383264] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 52.007168] Key type lgssc registered [ 52.898437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 60.935566] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 80.096369] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 86.141155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 86.153260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 87.289741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 87.304859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 87.365096] Lustre: lustre-MDT0000: new disk, initializing [ 87.398580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 87.410220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 89.281684] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 96.206805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 96.261220] Lustre: 6478:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 96.282224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 96.286239] Lustre: Skipped 1 previous similar message [ 96.340075] Lustre: lustre-MDT0001: new disk, initializing [ 96.367295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 96.376154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 96.379454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 98.223304] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.060052] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 105.624418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 105.780473] Lustre: lustre-OST0000: new disk, initializing [ 105.784078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 105.788743] Lustre: 8415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 105.818675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 108.456955] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.745985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 109.751918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 109.778722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 115.611140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 115.682974] Lustre: lustre-OST0001: new disk, initializing [ 115.686156] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 115.690720] Lustre: 9490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 115.719402] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 117.683984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 117.688922] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 117.703466] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 118.258601] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 125.241375] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 128.441447] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 131.082673] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing check_logdir /tmp/testlogs/ [ 133.006904] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing yml_node [ 134.964405] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 136.175793] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 137.388388] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 138.145350] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 30 18:09:59 EDT 2026 [ 145.722885] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 146.464446] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 158.688877] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 158.689159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 158.692874] Lustre: Skipped 2 previous similar messages [ 158.696480] Lustre: Skipped 2 previous similar messages [ 163.295891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 163.301056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 163.309440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 164.910642] Lustre: server umount lustre-MDT0000 complete [ 166.589371] LustreError: 7893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782857428 with bad export cookie 14764620294317470302 [ 166.592081] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 166.596927] LustreError: 7893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 168.928047] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 168.928678] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 168.936207] Lustre: Skipped 1 previous similar message [ 168.940235] Lustre: Skipped 3 previous similar messages [ 171.935925] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 171.939749] Lustre: Skipped 2 previous similar messages [ 172.799203] Lustre: server umount lustre-MDT0001 complete [ 180.679213] Lustre: server umount lustre-OST0000 complete [ 188.512710] Lustre: server umount lustre-OST0001 complete [ 191.369657] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 195.650912] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 212.837442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.973938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 212.994251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 213.052273] Lustre: lustre-MDT0000: new disk, initializing [ 213.085544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.094663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 214.963466] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 220.373081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.444614] Lustre: 15031:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.614197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 222.361330] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 227.058056] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 229.592264] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 232.763050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.917417] Lustre: lustre-OST0000: new disk, initializing [ 232.919808] Lustre: Skipped 1 previous similar message [ 232.922380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 232.925153] Lustre: Skipped 2 previous similar messages [ 232.928189] Lustre: 17001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.968211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.971842] Lustre: Skipped 1 previous similar message [ 234.702097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 234.707670] Lustre: Skipped 1 previous similar message [ 234.711341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 234.760205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 235.501178] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 240.224106] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 242.128960] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 242.245284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 244.084940] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 244.201073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 245.219465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 245.222129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 245.223504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 245.224554] LustreError: Skipped 1 previous similar message [ 245.232465] Lustre: Skipped 1 previous similar message [ 251.239033] Lustre: server umount lustre-OST0000 complete [ 255.458513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.459580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 255.466499] Lustre: Skipped 1 previous similar message [ 255.470457] Lustre: Skipped 3 previous similar messages [ 259.189689] Lustre: server umount lustre-MDT0000 complete [ 260.576605] LustreError: 15042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 260.586314] LustreError: 15042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 260.906296] LustreError: 15025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782857523 with bad export cookie 14764620294317471835 [ 260.913323] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 260.913808] LustreError: 15025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 261.057427] Lustre: server umount lustre-MDT0001 complete [ 262.980555] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:12:04 (1782857524) [ 264.112813] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 264.931518] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:12:06 (1782857526) [ 267.966468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 268.201281] LustreError: 19408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 268.241916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 270.087537] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 273.377121] LustreError: 19409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 274.045244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 276.094217] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 279.426770] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 281.878513] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 285.031605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 287.794492] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 291.160211] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 303.584420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 303.589712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.598417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 305.380719] Lustre: server umount lustre-OST0000 complete [ 307.217695] Lustre: server umount lustre-MDT0000 complete [ 308.887478] LustreError: 19388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782857571 with bad export cookie 14764620294317472717 [ 308.895097] LustreError: 19388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 308.896470] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.079602] Lustre: server umount lustre-MDT0001 complete [ 319.748314] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 321.225180] Key type lgssc unregistered [ 321.398694] LNet: 22958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.403645] LNetError: 22958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.415656] LNet: Removed LNI 192.168.204.113@tcp [ 321.917194] Key type .llcrypt unregistered [ 321.919466] Key type ._llcrypt unregistered [ 332.511968] Key type ._llcrypt registered [ 332.513918] Key type .llcrypt registered [ 332.570280] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 332.959442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 332.972258] alg: No test for adler32 (adler32-zlib) [ 333.858103] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 333.989799] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 335.599147] Key type lgssc registered [ 336.109908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 340.021330] loop: module loaded [ 340.052432] loop0: detected capacity change from 0 to 204800000 [ 340.130730] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.134709] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.294990] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 343.148165] loop0: detected capacity change from 0 to 204800000 [ 343.237201] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 343.242099] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 343.247787] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 345.608056] loop0: detected capacity change from 0 to 204800000 [ 348.132988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 348.142851] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 349.236877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 349.255099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 349.299647] Lustre: lustre-MDT0000: new disk, initializing [ 349.324254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.333397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 351.082940] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 355.694308] loop1: detected capacity change from 0 to 204800000 [ 358.211437] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 358.360115] Lustre: lustre-OST07c6: new disk, initializing [ 358.363492] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 358.367367] Lustre: Skipped 1 previous similar message [ 358.371264] Lustre: 26124:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 358.413447] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 361.090572] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 364.566411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 364.572694] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 364.590865] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 368.715459] Lustre: 25110:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 368.721611] Lustre: 25110:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 368.726350] Lustre: 25110:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 368.731378] Lustre: 25110:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 368.736610] Lustre: 25110:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 368.741277] Lustre: 25110:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 381.659929] Lustre: 25112:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 381.666126] Lustre: 25112:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 381.670998] Lustre: 25112:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 381.676029] Lustre: 25112:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 381.681365] Lustre: 25112:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 381.686086] Lustre: 25112:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 385.447228] Lustre: 25110:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 385.453366] Lustre: 25110:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 385.458128] Lustre: 25110:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 385.462925] Lustre: 25110:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 385.467872] Lustre: 25110:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 385.472829] Lustre: 25110:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 385.477817] Lustre: 25110:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 385.483126] Lustre: 25110:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 385.487990] Lustre: 25110:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 385.492636] Lustre: 25110:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 385.497535] Lustre: 25110:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 385.502239] Lustre: 25110:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 388.581508] Lustre: 26132:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 388.587569] Lustre: 26132:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 388.592399] Lustre: 26132:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 388.597124] Lustre: 26132:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 388.602126] Lustre: 26132:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 388.607213] Lustre: 26132:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 388.612191] Lustre: 26132:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 388.617598] Lustre: 26132:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 388.622606] Lustre: 26132:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 388.627491] Lustre: 26132:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 388.632490] Lustre: 26132:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 388.637397] Lustre: 26132:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 395.433400] Lustre: 25110:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 395.439607] Lustre: 25110:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 395.444543] Lustre: 25110:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 395.449374] Lustre: 25110:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 395.454490] Lustre: 25110:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 395.459543] Lustre: 25110:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 395.464530] Lustre: 25110:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 395.469914] Lustre: 25110:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 395.474889] Lustre: 25110:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 395.479692] Lustre: 25110:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 395.484623] Lustre: 25110:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 395.490093] Lustre: 25110:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 405.313432] Lustre: 26132:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 405.319682] Lustre: 26132:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 405.324704] Lustre: 26132:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 405.329646] Lustre: 26132:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 405.334775] Lustre: 26132:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 405.340177] Lustre: 26132:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 405.345270] Lustre: 26132:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 405.350738] Lustre: 26132:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 405.355792] Lustre: 26132:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 405.360681] Lustre: 26132:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 405.365775] Lustre: 26132:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 405.370672] Lustre: 26132:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 451.554669] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.554732] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 451.563674] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 456.673533] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 461.279152] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 461.354070] Lustre: server umount lustre-OST07c6 complete [ 465.763167] Lustre: server umount lustre-MDT0000 complete [ 472.970639] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 474.411928] Key type lgssc unregistered [ 474.576596] LNet: 28702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.581504] LNetError: 28702:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 474.592652] LNet: Removed LNI 192.168.204.113@tcp [ 475.052913] Key type .llcrypt unregistered [ 475.055162] Key type ._llcrypt unregistered [ 484.020701] Key type ._llcrypt registered [ 484.023022] Key type .llcrypt registered [ 484.078890] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 484.372379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 484.385801] alg: No test for adler32 (adler32-zlib) [ 485.281994] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 485.419448] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 487.031240] Key type lgssc registered [ 487.651440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 491.415619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 491.427425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.655558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.623590] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 498.671581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.863636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 500.773179] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 504.140553] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 506.601034] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 509.790181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 510.005935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 512.744944] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 516.095915] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 518.192678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 533.472566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 533.477240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.485615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 534.249220] Lustre: server umount lustre-OST0000 complete [ 539.619201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.621235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.627377] Lustre: Skipped 1 previous similar message [ 544.738181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 544.742152] Lustre: Skipped 2 previous similar messages [ 549.855215] Lustre: 29261:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782857796/real 1782857796] req@00000000ee2c0465 x1869461844943488/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782857812 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 549.858364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 549.871076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.874877] Lustre: Skipped 1 previous similar message [ 550.367130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 550.454887] Lustre: server umount lustre-MDT0000 complete [ 552.226695] LustreError: 29931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782857814 with bad export cookie 7676578544073461959 [ 552.234197] LustreError: 29931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 552.238432] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.384451] Lustre: server umount lustre-MDT0001 complete [ 557.175370] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 558.747831] Key type lgssc unregistered [ 558.918709] LNet: 33703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.924325] LNetError: 33703:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 558.936672] LNet: Removed LNI 192.168.204.113@tcp [ 559.404345] Key type .llcrypt unregistered [ 559.406781] Key type ._llcrypt unregistered [ 568.950315] Key type ._llcrypt registered [ 568.952341] Key type .llcrypt registered [ 569.018639] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 569.521466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 569.551204] alg: No test for adler32 (adler32-zlib) [ 570.462455] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 570.567884] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 572.175175] Key type lgssc registered [ 572.796941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 576.746391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 576.758388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.965032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.920377] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 584.024146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.240533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 586.068626] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 589.551890] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 592.249359] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 595.505637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 595.709815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 598.494816] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 601.968470] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 602.931122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 618.464330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 618.469360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 618.478082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 619.492076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 619.495577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 619.501101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.018376] Lustre: server umount lustre-OST0000 complete [ 625.120459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 625.125495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.134268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 625.138166] Lustre: Skipped 1 previous similar message [ 629.729804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 629.731267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 631.040359] Lustre: server umount lustre-MDT0000 complete [ 632.782378] LustreError: 37548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782857895 with bad export cookie 12545611270500775320 [ 632.786366] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 632.790117] LustreError: 37548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 632.945485] Lustre: server umount lustre-MDT0001 complete [ 637.888796] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 639.383934] Key type lgssc unregistered [ 639.554694] LNet: 38502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.559629] LNetError: 38502:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 639.575523] LNet: Removed LNI 192.168.204.113@tcp [ 640.022144] Key type .llcrypt unregistered [ 640.024490] Key type ._llcrypt unregistered [ 652.044270] Key type ._llcrypt registered [ 652.045976] Key type .llcrypt registered [ 652.100544] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 652.453386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 652.478203] alg: No test for adler32 (adler32-zlib) [ 653.364151] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 653.490637] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 655.087171] Key type lgssc registered [ 655.708869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 659.691750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 659.703735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 660.941094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.806588] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 666.950592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.162955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 669.079307] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 672.627822] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 675.265499] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 678.458581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 678.636376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.240233] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 684.626234] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 685.803978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 697.824023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 697.829055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.837777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 701.418662] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 702.435772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.437472] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.789281] Lustre: server umount lustre-MDT0000 complete [ 705.513289] LustreError: 39881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782857967 with bad export cookie 5663486258032450433 [ 705.520305] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 705.521392] LustreError: 39881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 706.524885] LustreError: 39894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.536343] LustreError: 39894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 706.539084] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.13@tcp (stopping) [ 706.546383] Lustre: Skipped 2 previous similar messages [ 707.552518] LustreError: 39896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.552964] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.570569] Lustre: Skipped 1 previous similar message [ 710.560176] LustreError: 39894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 710.560617] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 710.574320] Lustre: Skipped 1 previous similar message [ 711.755286] Lustre: server umount lustre-MDT0001 complete [ 730.284602] Lustre: server umount lustre-OST0000 complete [ 737.579449] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 739.094035] Key type lgssc unregistered [ 739.264658] LNet: 43779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.269641] LNetError: 43779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 739.280632] LNet: Removed LNI 192.168.204.113@tcp [ 739.734510] Key type .llcrypt unregistered [ 739.736878] Key type ._llcrypt unregistered [ 751.692641] Key type ._llcrypt registered [ 751.694338] Key type .llcrypt registered [ 751.753386] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 752.093588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 752.129392] alg: No test for adler32 (adler32-zlib) [ 753.039978] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 753.158604] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 754.743180] Key type lgssc registered [ 755.360235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 759.322149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 759.333977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.542262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 762.462589] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 766.466408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 766.665594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 768.521919] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 771.948866] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 774.608263] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 777.803635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 777.966520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 780.584074] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 784.039453] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 785.132583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 797.669302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 797.677718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 797.678260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 797.686573] Lustre: Skipped 1 previous similar message [ 798.883410] Lustre: server umount lustre-OST0000 complete [ 812.583215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 812.584884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 812.591043] Lustre: Skipped 2 previous similar messages [ 812.597081] Lustre: Skipped 1 previous similar message [ 817.632833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 817.636930] Lustre: Skipped 1 previous similar message [ 818.926955] Lustre: server umount lustre-MDT0000 complete [ 820.672328] LustreError: 45154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782858083 with bad export cookie 3976366001168979446 [ 820.678439] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 820.679826] LustreError: 45154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 826.902144] Lustre: server umount lustre-MDT0001 complete [ 831.824748] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 833.264381] Key type lgssc unregistered [ 833.435537] LNet: 48900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.440523] LNetError: 48900:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.450548] LNet: Removed LNI 192.168.204.113@tcp [ 833.889370] Key type .llcrypt unregistered [ 833.891635] Key type ._llcrypt unregistered [ 846.040687] Key type ._llcrypt registered [ 846.042107] Key type .llcrypt registered [ 846.093915] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 846.412964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 846.455337] alg: No test for adler32 (adler32-zlib) [ 847.347591] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 847.470333] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 849.079147] Key type lgssc registered [ 849.713751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 853.651948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 853.664110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 854.874565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 856.757792] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 860.899609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.069214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 862.962786] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 866.492300] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 869.018614] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 872.304434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 872.525961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.299767] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 878.717620] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 879.729346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 891.435987] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 893.034576] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 903.994551] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-3018h22m45s [ 904.717452] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 905.702310] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 906.716600] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 906.722249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.730693] Lustre: Skipped 2 previous similar messages [ 907.746114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 907.750272] Lustre: Skipped 5 previous similar messages [ 911.984519] Lustre: server umount lustre-MDT0000 complete [ 912.867279] LustreError: 52702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 912.878522] LustreError: 52702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 913.594901] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 913.891510] LustreError: 50289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 913.901749] LustreError: 50289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 916.714423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.783756] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 916.932720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.980584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 917.989558] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 918.738671] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 920.367432] Lustre: DEBUG MARKER: Wait for df (36438) ... [ 921.110221] Lustre: DEBUG MARKER: done [ 923.109407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.112452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 923.117555] Lustre: Skipped 1 previous similar message [ 923.124306] Lustre: Skipped 1 previous similar message [ 928.228775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 928.232831] Lustre: Skipped 1 previous similar message [ 928.417769] Lustre: server umount lustre-OST0000 complete [ 933.346599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.347182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 933.354648] Lustre: Skipped 1 previous similar message [ 936.306116] Lustre: server umount lustre-MDT0000 complete [ 938.063077] LustreError: 50274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782858200 with bad export cookie 8758938508226023958 [ 938.070613] LustreError: 50274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 938.071169] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 938.224862] Lustre: server umount lustre-MDT0001 complete [ 943.058935] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 944.508165] Key type lgssc unregistered [ 944.675667] LNet: 55230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.680910] LNetError: 55230:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.690529] LNet: Removed LNI 192.168.204.113@tcp [ 945.125511] Key type .llcrypt unregistered [ 945.127724] Key type ._llcrypt unregistered [ 963.791634] Key type ._llcrypt registered [ 963.792847] Key type .llcrypt registered [ 963.839614] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 964.264633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 964.296771] alg: No test for adler32 (adler32-zlib) [ 965.212145] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 965.318494] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 966.919202] Key type lgssc registered [ 967.529265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 971.336555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 971.348414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 972.480037] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 972.488869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 972.564331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 974.366134] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 978.533656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.589899] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 978.610400] Lustre: 57416:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 978.767773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 980.691613] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 984.105610] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 986.541439] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 989.673285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 989.768738] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 989.868667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.465367] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 995.836486] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 995.879114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 999.708895] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-3018h24m20s [ 1000.471322] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1003.759453] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1004.918570] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1004.941826] LustreError: 57424:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1025.012209] Lustre: lustre-MDT0000: Client 7831b513-6d5d-448e-96ca-2fe04ddca1b3 (at 192.168.204.13@tcp) reconnecting [ 1044.991145] LustreError: 57424:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1046.004870] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1047.174116] Lustre: DEBUG MARKER: done [ 1051.106467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1051.111604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.120354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1052.130141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1052.133997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1052.134617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.147107] Lustre: Skipped 1 previous similar message [ 1055.332539] Lustre: server umount lustre-OST0000 complete [ 1057.249487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.250473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1057.257754] Lustre: Skipped 1 previous similar message [ 1063.284677] Lustre: server umount lustre-MDT0000 complete [ 1065.022855] LustreError: 57409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782858327 with bad export cookie 17974229407269351776 [ 1065.029705] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1065.030311] LustreError: 57409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1065.183664] Lustre: server umount lustre-MDT0001 complete [ 1069.891619] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1071.339480] Key type lgssc unregistered [ 1071.506517] LNet: 61740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.511629] LNetError: 61740:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.520646] LNet: Removed LNI 192.168.204.113@tcp [ 1071.947349] Key type .llcrypt unregistered [ 1071.948800] Key type ._llcrypt unregistered [ 1093.083967] Key type ._llcrypt registered [ 1093.085077] Key type .llcrypt registered [ 1093.120164] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1093.566229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1093.574233] alg: No test for adler32 (adler32-zlib) [ 1094.451181] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 1094.552279] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1096.151142] Key type lgssc registered [ 1096.709131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1098.443316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1098.454701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.585754] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1099.596049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1099.689641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.158604] Lustre: Failing over lustre-MDT0000 [ 1100.314386] Lustre: server umount lustre-MDT0000 complete [ 1103.422420] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 18:26:04 (1782858364) [ 1106.609200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1106.817171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.688670] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1112.817770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.866672] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1112.882506] Lustre: 64511:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1113.034466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1114.959164] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1118.449371] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1121.026308] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1124.289206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1124.403901] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1124.510548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1126.950613] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1130.461238] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1132.608461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1139.426875] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1142.757155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.759709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.764775] Lustre: Skipped 2 previous similar messages [ 1142.768821] Lustre: Skipped 1 previous similar message [ 1147.874539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1147.878399] Lustre: Skipped 2 previous similar messages [ 1148.035807] Lustre: server umount lustre-MDT0000 complete [ 1149.810123] LustreError: 66377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782858412 with bad export cookie 1446982291435887461 [ 1149.817720] LustreError: 66377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1149.817748] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1149.991844] Lustre: server umount lustre-MDT0001 complete [ 1151.651960] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1153.865690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1157.957594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.031131] LustreError: 68266:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1160.036597] LustreError: 68266:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000003815f23a x1869462483534464/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1782858422 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1160.049725] LustreError: 68266:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1160.159968] LustreError: 63436:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000085c44c2a x1869462483535104/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1160.411389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.463422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1162.332087] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1165.796051] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.798274] LustreError: 66361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1165.813969] LustreError: 66361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1166.418950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.600465] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1169.631196] Lustre: 63439:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782858415/real 1782858415] req@00000000e303020c x1869462483534720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782858431 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1171.615173] Lustre: 63437:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782858417/real 1782858417] req@00000000a1da39ea x1869462483534848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782858433 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1171.949896] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1171.982781] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1172.959279] Lustre: 63438:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782858419/real 1782858419] req@00000000ebdd87e8 x1869462483534976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782858435 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1172.969073] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1174.574377] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1176.031082] Lustre: 63439:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782858422/real 1782858422] req@00000000d24515ee x1869462483535232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782858438 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1177.057275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1177.062158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1177.070635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.205072] LustreError: 66360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.212574] LustreError: 66360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1183.228876] Lustre: server umount lustre-MDT0000 complete [ 1185.036164] LustreError: 64505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782858447 with bad export cookie 1446982291435889729 [ 1185.043307] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1185.043666] LustreError: 64505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1185.213276] Lustre: server umount lustre-MDT0001 complete [ 1188.291786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1192.789857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.911240] LustreError: 70943:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1194.916715] LustreError: 70943:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000d1652d59 x1869462483555584/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1782858457 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1194.929963] LustreError: 70943:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1195.488576] LustreError: 63436:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006cea1fd5 x1869462483556224/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1195.659625] LustreError: 69009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1195.697890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.701245] Lustre: Skipped 1 previous similar message [ 1195.735397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1197.626767] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1201.124969] LustreError: 66360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.127842] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.143566] Lustre: Skipped 2 previous similar messages [ 1201.795604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.958192] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1203.999173] Lustre: 63437:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782858450/real 1782858450] req@00000000a4e3c948 x1869462483555840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782858466 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1207.277982] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1207.418250] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1210.035344] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1213.411868] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.415186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1213.419530] Lustre: Skipped 1 previous similar message [ 1213.423589] Lustre: Skipped 3 previous similar messages [ 1218.531049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1218.535074] Lustre: Skipped 3 previous similar messages [ 1218.683993] Lustre: server umount lustre-MDT0000 complete [ 1220.460818] LustreError: 64505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782858482 with bad export cookie 1446982291435891108 [ 1220.465116] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1220.468445] LustreError: 64505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1220.614681] Lustre: server umount lustre-MDT0001 complete [ 1223.686867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1225.335878] Lustre: DEBUG MARKER: files compared the same [ 1236.611849] Lustre: server umount lustre-OST0000 complete [ 1244.368891] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1245.960821] Key type lgssc unregistered [ 1246.149700] LNet: 74225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.154584] LNetError: 74225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.169835] LNet: Removed LNI 192.168.204.113@tcp [ 1246.659715] Key type .llcrypt unregistered [ 1246.661912] Key type ._llcrypt unregistered [ 1258.691888] Key type ._llcrypt registered [ 1258.693480] Key type .llcrypt registered [ 1258.760798] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1259.141945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1259.170035] alg: No test for adler32 (adler32-zlib) [ 1260.060124] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 1260.163668] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1261.743178] Key type lgssc registered [ 1262.317762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1266.221986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1266.234161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.459875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.406875] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug +malloc all [ 1273.532846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.667380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1275.672306] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug +malloc all [ 1279.186962] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1281.882533] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1285.206792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1285.372775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1288.160584] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug +malloc all [ 1291.617794] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1292.580667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1308.127878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1308.132817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.141257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1309.153154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.153301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1309.161540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1309.165427] Lustre: Skipped 1 previous similar message [ 1310.624821] Lustre: server umount lustre-OST0000 complete [ 1314.272116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1314.280756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1314.284701] Lustre: Skipped 1 previous similar message [ 1318.569678] Lustre: server umount lustre-MDT0000 complete [ 1319.392738] LustreError: 75620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.402922] LustreError: 75620:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1320.290403] LustreError: 75601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782858582 with bad export cookie 4787168203938974206 [ 1320.292912] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1320.297972] LustreError: 75601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1320.440473] Lustre: server umount lustre-MDT0001 complete [ 1325.285703] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1326.667438] Key type lgssc unregistered [ 1326.850627] LNet: 79176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.855536] LNetError: 79176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.867658] LNet: Removed LNI 192.168.204.113@tcp [ 1327.300371] Key type .llcrypt unregistered [ 1327.302713] Key type ._llcrypt unregistered [ 1339.395037] Key type ._llcrypt registered [ 1339.396775] Key type .llcrypt registered [ 1339.454707] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1339.750226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1339.762762] alg: No test for adler32 (adler32-zlib) [ 1340.681748] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 1340.808691] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1342.399165] Key type lgssc registered [ 1342.959502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1346.881553] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1346.893623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1362.335407] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1367.455381] LustreError: 80543:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [ 1394.079368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.805885] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1400.334646] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1404.000765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.063641] Lustre: *** cfs_fail_loc=706, val=0*** [ 1404.266293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.315476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1406.039076] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1409.380823] LustreError: 80569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.391129] LustreError: 80569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1410.005696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.217878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1411.900289] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1415.224548] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1417.742844] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1419.744372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1419.749419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.757963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1420.773586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1420.773673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1420.778553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.894337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1425.898388] Lustre: Skipped 2 previous similar messages [ 1431.012650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1431.016667] Lustre: Skipped 1 previous similar message [ 1433.055179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1433.131283] Lustre: server umount lustre-OST0000 complete [ 1436.130881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1436.131263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.136783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1436.143752] Lustre: Skipped 1 previous similar message [ 1441.069500] Lustre: server umount lustre-MDT0000 complete [ 1441.248859] LustreError: 81630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.259303] LustreError: 81630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1442.791976] LustreError: 80549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782858705 with bad export cookie 5079308947228958056 [ 1442.797540] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1442.799634] LustreError: 80549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1442.951282] Lustre: server umount lustre-MDT0001 complete [ 1447.640613] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1449.206982] Key type lgssc unregistered [ 1449.389643] LNet: 84157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.394390] LNetError: 84157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1449.404503] LNet: Removed LNI 192.168.204.113@tcp [ 1449.878541] Key type .llcrypt unregistered [ 1449.880930] Key type ._llcrypt unregistered [ 1461.993814] Key type ._llcrypt registered [ 1461.996312] Key type .llcrypt registered [ 1462.064840] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1462.431086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1462.453180] alg: No test for adler32 (adler32-zlib) [ 1463.368245] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 1463.500279] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1465.095145] Key type lgssc registered [ 1465.705634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1469.581243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1469.592710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1472.583994] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1476.256482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1476.556237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1478.463559] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1481.698050] LustreError: 86237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1481.708723] LustreError: 86237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1482.540693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1482.734992] LustreError: 86235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.745363] LustreError: 86235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1482.750628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.495339] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1485.860748] LustreError: 86240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1485.871704] LustreError: 86240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1489.538611] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1489.548240] LustreError: 84909:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c429d26d x1869462870434560/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1489.563202] LustreError: 86241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.565467] Lustre: MGS: Client 4286b842-0891-4159-ae8a-3d78284c65ab (at 0@lo) reconnecting [ 1489.573511] LustreError: 86241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1489.596079] LustreError: 86251:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1489.726350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1489.786334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1491.603665] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1501.152547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1501.157453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.166783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1505.250825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1505.254269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1505.256034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.534146] Lustre: server umount lustre-OST0000 complete [ 1510.370397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1510.371440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.384976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1510.388838] Lustre: Skipped 2 previous similar messages [ 1514.388276] Lustre: server umount lustre-MDT0000 complete [ 1515.489917] LustreError: 88150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1515.500255] LustreError: 88150:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1516.270903] Lustre: server umount lustre-MDT0001 complete [ 1518.011256] Lustre: server umount MGS complete [ 1523.199926] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:33:04 (1782858784) [ 1536.083537] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 1540.529773] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1557.609727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1559.621279] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1564.830976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.974745] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1565.007714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1565.064779] Lustre: lustre-MDT0001: new disk, initializing [ 1565.106879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1565.123152] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1565.127913] Lustre: 93399:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1565.133482] LustreError: 93399:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1567.101754] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1572.721337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1572.868618] Lustre: lustre-OST0000: new disk, initializing [ 1572.872319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1572.878163] Lustre: 94165:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1572.930074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1575.651409] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1578.976586] Lustre: 94172:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000006fb024a6 x1869462870461696/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782858857 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1580.478283] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1583.713511] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1583.722672] Lustre: MGS: Client 6bddf3e7-1c32-4d69-91df-892daa9ed510 (at 0@lo) reconnecting [ 1583.750946] Lustre: 95212:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1583.769330] LustreError: 93384:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1583.775494] LustreError: 93384:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1583.785263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1583.838861] Lustre: lustre-MDT0000: new disk, initializing [ 1583.911437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1583.927407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1584.991767] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1584.996624] Lustre: 95238:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000a4aa56c2 x1869462870467712/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1782858863 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1585.903564] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1590.051507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1590.057483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1590.125390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1613.798878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1613.804153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1613.806940] Lustre: Skipped 2 previous similar messages [ 1613.814114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1618.915183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1618.919304] Lustre: Skipped 2 previous similar messages [ 1625.055165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1625.138541] Lustre: server umount lustre-OST0000 complete [ 1629.156213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1629.156666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.162324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1629.169073] Lustre: Skipped 1 previous similar message [ 1629.173051] Lustre: Skipped 2 previous similar messages [ 1633.242253] Lustre: server umount lustre-MDT0000 complete [ 1634.273978] LustreError: 95667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.287783] LustreError: 95667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1641.318728] Lustre: server umount lustre-MDT0001 complete [ 1643.148485] Lustre: server umount MGS complete [ 1648.512932] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:35:09 (1782858909) [ 1656.837850] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1658.405422] Key type lgssc unregistered [ 1658.584700] LNet: 97736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.589715] LNetError: 97736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.603645] LNet: Removed LNI 192.168.204.113@tcp [ 1659.102395] Key type .llcrypt unregistered [ 1659.104686] Key type ._llcrypt unregistered [ 1668.207847] Key type ._llcrypt registered [ 1668.209088] Key type .llcrypt registered [ 1668.263669] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1668.608631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1668.693414] alg: No test for adler32 (adler32-zlib) [ 1669.605566] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 1669.725138] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1671.327146] Key type lgssc registered [ 1671.955231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1676.016382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1676.028444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.259533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1679.270337] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1683.504519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.724113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1685.551779] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1688.928590] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1691.404192] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1694.520639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1694.710375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.263666] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1700.569400] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1701.867578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1714.149050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.149575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.158420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1714.166111] Lustre: Skipped 1 previous similar message [ 1715.749577] Lustre: server umount lustre-OST0000 complete [ 1717.592184] Lustre: server umount lustre-MDT0000 complete [ 1719.266863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1719.268192] LustreError: 98931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1719.275146] Lustre: Skipped 2 previous similar messages [ 1719.288924] LustreError: 98931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1719.307191] LustreError: 98916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782858981 with bad export cookie 18270142455629984982 [ 1719.313044] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1719.314771] LustreError: 98916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1725.517142] Lustre: server umount lustre-MDT0001 complete [ 1730.423453] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1731.869057] Key type lgssc unregistered [ 1732.030700] LNet: 102485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.035661] LNetError: 102485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.047452] LNet: Removed LNI 192.168.204.113@tcp [ 1732.470766] Key type .llcrypt unregistered [ 1732.472343] Key type ._llcrypt unregistered [ 1741.078336] Key type ._llcrypt registered [ 1741.080329] Key type .llcrypt registered [ 1741.146640] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1741.552898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1741.560466] alg: No test for adler32 (adler32-zlib) [ 1742.467431] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 1742.600360] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1744.207155] Key type lgssc registered [ 1744.768168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1747.794055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1747.794163] LustreError: 103546:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1747.798302] Lustre: Skipped 1 previous similar message [ 1747.805452] LustreError: 103547:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1747.809715] LustreError: 103546:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 1747.814168] LustreError: 103546:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1747.818296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1747.820584] LustreError: 103546:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1749.005471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1752.649992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1752.798876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1754.645458] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1756.901205] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing lsmod [ 1758.271918] LustreError: 104752:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1758.271990] LustreError: 104753:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1758.280121] LustreError: 104752:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1758.284605] LustreError: 104752:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1758.290871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1758.291092] LustreError: 104752:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1758.450537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.468138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1762.851903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1762.855433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1762.860725] Lustre: Skipped 1 previous similar message [ 1767.906436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1767.910522] Lustre: Skipped 4 previous similar messages [ 1773.026034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1773.030109] Lustre: Skipped 2 previous similar messages [ 1774.047154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1774.125448] Lustre: server umount lustre-MDT0000 complete [ 1775.869802] LustreError: 104791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782859038 with bad export cookie 8955738395056920758 [ 1775.875655] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1775.877443] LustreError: 104791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1777.183311] LustreError: 104017:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008a477e51 x1869463162997888/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1777.259400] Lustre: server umount lustre-MDT0001 complete [ 1789.301834] Lustre: server umount lustre-OST0000 complete [ 1793.262309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.464789] LustreError: 105818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1793.474051] LustreError: 105818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1793.497275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1795.382971] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1798.625560] LustreError: 105819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1799.377656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.562938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1801.448886] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1804.836777] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1807.364155] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1810.571757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1810.755490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1813.476816] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1816.999541] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1817.969856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 1833.441848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1833.446836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.454765] Lustre: Skipped 1 previous similar message [ 1833.458390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1834.470344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1834.475391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1836.713387] Lustre: server umount lustre-OST0000 complete [ 1839.587369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.588802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1839.595190] Lustre: Skipped 1 previous similar message [ 1839.599275] Lustre: Skipped 2 previous similar messages [ 1844.707723] LustreError: 105813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1844.715841] LustreError: 105813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1844.731269] Lustre: server umount lustre-MDT0000 complete [ 1846.525307] LustreError: 105800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782859108 with bad export cookie 8955738395056921675 [ 1846.532979] LustreError: 105800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1846.534901] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1846.695591] Lustre: server umount lustre-MDT0001 complete [ 1851.724183] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1853.354977] Key type lgssc unregistered [ 1853.546577] LNet: 109371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.551537] LNetError: 109371:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1853.562597] LNet: Removed LNI 192.168.204.113@tcp [ 1854.029299] Key type .llcrypt unregistered [ 1854.031437] Key type ._llcrypt unregistered [ 1866.061962] Key type ._llcrypt registered [ 1866.063828] Key type .llcrypt registered [ 1866.125599] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1866.431518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1866.462052] alg: No test for adler32 (adler32-zlib) [ 1867.375355] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 1867.505857] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1869.119141] Key type lgssc registered [ 1869.751448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1873.689180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1873.700881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1874.937909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.949312] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1881.137077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1881.314069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1883.242461] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1886.817756] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1889.442796] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1892.754645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1892.945338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.789653] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1899.242089] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1900.150060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1911.598643] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1912.650606] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1915.872500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1915.877551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.885930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1916.901435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1916.904966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1916.906569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.169958] Lustre: server umount lustre-OST0000 complete [ 1922.019231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.020114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1922.026988] Lustre: Skipped 1 previous similar message [ 1922.030298] Lustre: Skipped 1 previous similar message [ 1927.137513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.141635] Lustre: Skipped 2 previous similar messages [ 1932.257923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1932.261364] Lustre: Skipped 1 previous similar message [ 1936.351162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1936.439634] Lustre: server umount lustre-MDT0000 complete [ 1937.377499] LustreError: 110765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1937.387616] LustreError: 110765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1938.255713] LustreError: 110745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782859200 with bad export cookie 12632111669402000421 [ 1938.260740] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1938.263176] LustreError: 110745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1938.437448] Lustre: server umount lustre-MDT0001 complete [ 1943.297362] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1944.887744] Key type lgssc unregistered [ 1945.067774] LNet: 114423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.072865] LNetError: 114423:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.083749] LNet: Removed LNI 192.168.204.113@tcp [ 1945.582127] Key type .llcrypt unregistered [ 1945.584407] Key type ._llcrypt unregistered [ 1954.782337] Key type ._llcrypt registered [ 1954.784045] Key type .llcrypt registered [ 1954.844402] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1955.363608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.378470] alg: No test for adler32 (adler32-zlib) [ 1956.265117] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 1956.382901] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1957.983161] Key type lgssc registered [ 1958.622903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1962.647253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1962.658866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.882445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.876099] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1970.056978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.273505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1972.173147] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1975.677087] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1978.357994] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1981.658293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1981.808050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.434396] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1987.770526] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1988.972479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2004.448429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2004.450569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.453713] LustreError: Skipped 1 previous similar message [ 2004.462613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2005.474917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2005.475215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2005.481522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.142086] Lustre: server umount lustre-OST0000 complete [ 2010.594447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.595464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.602148] Lustre: Skipped 1 previous similar message [ 2010.605928] Lustre: Skipped 1 previous similar message [ 2015.713277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.717278] Lustre: Skipped 2 previous similar messages [ 2020.832906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2020.837467] Lustre: Skipped 1 previous similar message [ 2023.391147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2023.473157] Lustre: server umount lustre-MDT0000 complete [ 2024.929589] LustreError: 115622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.939823] LustreError: 115622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2025.189666] LustreError: 115603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782859287 with bad export cookie 18317353470287434277 [ 2025.196611] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.197325] LustreError: 115603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2025.348290] Lustre: server umount lustre-MDT0001 complete [ 2030.059039] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2031.652274] Key type lgssc unregistered [ 2031.830655] LNet: 119178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.835779] LNetError: 119178:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2031.851751] LNet: Removed LNI 192.168.204.113@tcp [ 2032.331519] Key type .llcrypt unregistered [ 2032.333909] Key type ._llcrypt unregistered [ 2044.811755] Key type ._llcrypt registered [ 2044.813479] Key type .llcrypt registered [ 2044.874225] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2045.298579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2045.351052] alg: No test for adler32 (adler32-zlib) [ 2046.247738] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 2046.384187] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2047.999138] Key type lgssc registered [ 2048.636406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2052.585715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2052.596908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.785684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.743361] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2059.896199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.105887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2062.125438] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2065.745407] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2068.390524] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2071.674186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2071.880185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2074.596574] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2078.088392] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2079.090124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 2091.571558] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2093.352884] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2093.357600] Lustre: Skipped 2 previous similar messages [ 2095.405908] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2095.410717] Lustre: Skipped 2 previous similar messages [ 2098.711921] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2098.737321] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2105.741664] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.13@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2105.748924] Lustre: Skipped 2 previous similar messages [ 2111.503841] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.13@tcp 192.168.0.[2,10]@tcp [ 2113.488874] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.13/24@tcp in log lustre-MDT0000 [ 2113.494498] Lustre: Skipped 2 previous similar messages [ 2123.272584] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.13/24@tcp [ 2123.276296] Lustre: Skipped 1 previous similar message [ 2126.308781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2126.313310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2126.314927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2126.321901] Lustre: Skipped 1 previous similar message [ 2131.428688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2131.432673] Lustre: Skipped 2 previous similar messages [ 2131.690575] Lustre: server umount lustre-OST0000 complete [ 2136.547673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.548982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2136.555329] Lustre: Skipped 1 previous similar message [ 2139.698979] Lustre: server umount lustre-MDT0000 complete [ 2141.497905] LustreError: 120554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782859403 with bad export cookie 12642804266660921834 [ 2141.505574] LustreError: 120554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2141.506265] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2141.678266] Lustre: server umount lustre-MDT0001 complete [ 2146.875155] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2148.490374] Key type lgssc unregistered [ 2148.679877] LNet: 125814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2148.684970] LNetError: 125814:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.694822] LNet: Removed LNI 192.168.204.113@tcp [ 2149.161858] Key type .llcrypt unregistered [ 2149.164152] Key type ._llcrypt unregistered [ 2163.495297] Key type ._llcrypt registered [ 2163.497144] Key type .llcrypt registered [ 2163.563845] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2163.929072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2163.938643] alg: No test for adler32 (adler32-zlib) [ 2164.857724] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 2164.994144] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2166.599173] Key type lgssc registered [ 2167.226369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2171.214869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2171.226724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.386814] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2172.396864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.396912] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.13/24@tcp [ 2174.385485] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2178.575358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.743344] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2178.756375] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.13/24@tcp [ 2178.783810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2180.741869] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2184.322166] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2186.968584] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2190.280949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2190.495132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.210540] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2196.659915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 2196.772204] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2212.320377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2212.325498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.334262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2214.374673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2214.376077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2214.379402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.745950] Lustre: server umount lustre-OST0000 complete [ 2219.491894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2219.492952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2219.500059] Lustre: Skipped 1 previous similar message [ 2219.503930] Lustre: Skipped 1 previous similar message [ 2224.609686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2224.613733] Lustre: Skipped 2 previous similar messages [ 2224.772305] Lustre: server umount lustre-MDT0000 complete [ 2226.594850] LustreError: 127331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782859488 with bad export cookie 10199276936204694154 [ 2226.602593] LustreError: 127331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2226.604073] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2226.763781] Lustre: server umount lustre-MDT0001 complete [ 2231.765868] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2233.363934] Key type lgssc unregistered [ 2233.550113] LNet: 130960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2233.555371] LNetError: 130960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2233.566674] LNet: Removed LNI 192.168.204.113@tcp [ 2234.030540] Key type .llcrypt unregistered [ 2234.032161] Key type ._llcrypt unregistered [ 2246.216756] Key type ._llcrypt registered [ 2246.218316] Key type .llcrypt registered [ 2246.279889] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 2254.398820] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2254.895792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2254.906048] alg: No test for adler32 (adler32-zlib) [ 2255.787724] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 2255.900833] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2257.511181] Key type lgssc registered [ 2258.159293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2275.220602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2275.231898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.354624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2276.371680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2276.424857] Lustre: lustre-MDT0000: new disk, initializing [ 2276.453455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.464623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2278.469808] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2284.080151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.144197] Lustre: 134334:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2284.186682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2284.190396] Lustre: Skipped 1 previous similar message [ 2284.234330] Lustre: lustre-MDT0001: new disk, initializing [ 2284.286391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2284.309153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2284.316696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2286.173102] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2290.954179] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2293.563973] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2296.856897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2297.042084] Lustre: lustre-OST0000: new disk, initializing [ 2297.045234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2297.049824] Lustre: 136303:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2297.103685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2299.900956] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2302.478364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2302.485457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2302.546875] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2304.736758] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2306.739781] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2306.861270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2308.872554] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2309.003380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2312.678862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2312.680288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2312.683949] LustreError: Skipped 1 previous similar message [ 2312.693151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2312.698863] Lustre: Skipped 1 previous similar message [ 2316.078231] Lustre: server umount lustre-OST0000 complete [ 2320.352527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2320.357644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.365454] Lustre: Skipped 1 previous similar message [ 2320.369245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2322.913509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2322.914983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.087670] Lustre: server umount lustre-MDT0000 complete [ 2325.910693] LustreError: 134327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782859588 with bad export cookie 6600634081365237785 [ 2325.915754] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2325.918866] LustreError: 134327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2326.084354] Lustre: server umount lustre-MDT0001 complete [ 2330.221644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2330.517650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.517459] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2335.714813] LustreError: 138516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.725102] LustreError: 138516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2336.626994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2336.841806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2338.805945] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2342.206310] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2344.740936] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2348.034332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2348.215955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.945666] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2354.357724] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2356.427693] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2356.562053] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2358.568899] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2362.830746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 2367.210616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2367.325563] Lustre: lustre-OST0001: new disk, initializing [ 2367.328507] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2367.333186] Lustre: 141913:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2367.391514] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2370.207455] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2374.573516] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2377.246607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2377.253906] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2377.321604] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2377.803196] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2379.769995] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2379.904407] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2382.458171] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c8c90f50800.ost_server_uuid 50 [ 2383.147147] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c8c90f50800.ost_server_uuid in FULL state after 0 sec [ 2387.428310] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2387.433286] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.435797] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2387.440858] Lustre: Skipped 1 previous similar message [ 2387.444926] Lustre: Skipped 1 previous similar message [ 2391.757241] Lustre: server umount lustre-OST0001 complete [ 2392.549347] LustreError: 140335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.559805] LustreError: 140335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2393.568500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2393.573265] LustreError: Skipped 1 previous similar message [ 2393.575995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.584577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2397.674955] LustreError: 140335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.685320] LustreError: 140335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2399.722046] Lustre: server umount lustre-OST0000 complete [ 2402.787397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.789111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2402.795496] Lustre: Skipped 2 previous similar messages [ 2402.799451] Lustre: Skipped 3 previous similar messages [ 2407.739612] Lustre: server umount lustre-MDT0000 complete [ 2407.904827] LustreError: 138515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.915155] LustreError: 138515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2409.530070] LustreError: 141912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782859671 with bad export cookie 6600634081365238653 [ 2409.537495] LustreError: 141912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2409.540673] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2409.697929] Lustre: server umount lustre-MDT0001 complete [ 2418.429924] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2419.925654] Key type lgssc unregistered [ 2420.099636] LNet: 144931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2420.104799] LNetError: 144931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.114512] LNet: Removed LNI 192.168.204.113@tcp [ 2420.573242] Key type .llcrypt unregistered [ 2420.575515] Key type ._llcrypt unregistered [ 2438.597883] Key type ._llcrypt registered [ 2438.599539] Key type .llcrypt registered [ 2438.645065] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 2446.550768] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2446.941561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2446.961388] alg: No test for adler32 (adler32-zlib) [ 2447.858331] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 2447.972980] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2449.567161] Key type lgssc registered [ 2450.205472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2467.202887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2467.213619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2468.328106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2468.345278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2468.404643] Lustre: lustre-MDT0000: new disk, initializing [ 2468.436750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2468.449124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2470.287243] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2475.802476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.874009] Lustre: 149057:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2475.912040] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2475.916078] Lustre: Skipped 1 previous similar message [ 2475.964202] Lustre: lustre-MDT0001: new disk, initializing [ 2476.011125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2476.029936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2476.036156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2477.809113] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2482.577312] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2485.095898] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2488.366118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2488.545573] Lustre: lustre-OST0000: new disk, initializing [ 2488.549559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2488.555966] Lustre: 151024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2488.606350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.955430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2488.965222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2489.014072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2491.198875] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2496.028220] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2498.493968] Lustre: Failing over lustre-OST0000 [ 2498.596308] Lustre: server umount lustre-OST0000 complete [ 2512.677276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2512.821718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2512.828352] Lustre: Skipped 1 previous similar message [ 2512.831717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.842242] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2512.857563] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2514.848169] Lustre: 147284:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782859761/real 1782859761] req@00000000db61a2ed x1869463903300352/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782859777 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2515.472260] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2517.266840] Lustre: Failing over lustre-MDT0000 [ 2517.387243] Lustre: server umount lustre-MDT0000 complete [ 2517.986287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.995387] Lustre: Skipped 2 previous similar messages [ 2521.055150] Lustre: 147286:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782859767/real 1782859767] req@00000000a64b6d3b x1869463903302272/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782859783 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2521.070795] Lustre: 147286:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2523.101411] LustreError: 149064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2523.112934] LustreError: 149064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2526.175138] Lustre: 147284:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782859772/real 1782859772] req@0000000058976b6b x1869463903303424/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782859788 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2528.221442] LustreError: 149063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.232467] LustreError: 149063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2531.512842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.578426] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2531.584566] Lustre: 149061:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.113@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2531.683087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2531.708421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2531.747518] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2533.341265] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2533.508032] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2536.938857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2536.950477] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2536.984958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2542.054737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.055825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2542.056993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2542.062743] Lustre: Skipped 1 previous similar message [ 2542.067669] LustreError: Skipped 1 previous similar message [ 2544.356464] Lustre: server umount lustre-OST0000 complete [ 2547.169739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2547.171278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.184159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2547.188109] Lustre: Skipped 2 previous similar messages [ 2552.288684] LustreError: 149068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.296353] LustreError: 149068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2552.308302] Lustre: server umount lustre-MDT0000 complete [ 2554.021592] LustreError: 149050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782859816 with bad export cookie 6682846430740716637 [ 2554.028486] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2554.029087] LustreError: 149050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2554.175252] Lustre: server umount lustre-MDT0001 complete [ 2558.809553] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2560.350875] Key type lgssc unregistered [ 2560.535646] LNet: 154673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2560.540655] LNetError: 154673:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2560.551822] LNet: Removed LNI 192.168.204.113@tcp [ 2561.050137] Key type .llcrypt unregistered [ 2561.052401] Key type ._llcrypt unregistered [ 2584.867988] Key type ._llcrypt registered [ 2584.869958] Key type .llcrypt registered [ 2584.937996] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2585.321791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2585.335544] alg: No test for adler32 (adler32-zlib) [ 2586.205555] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 2586.318240] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2587.935155] Key type lgssc registered [ 2588.568773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2592.510857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2592.522679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.731584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.705545] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2600.674671] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2603.130811] LustreError: 156218:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2603.138105] LustreError: 156218:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2603.144332] LustreError: 156218:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2603.318439] Lustre: server umount lustre-MDT0000 complete [ 2610.353252] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 2615.009510] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2632.987475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.126221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2633.144911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2633.195635] Lustre: lustre-MDT0000: new disk, initializing [ 2633.229575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2633.241085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2635.265684] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2640.900160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.972471] Lustre: 159519:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2641.019966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2641.023908] Lustre: Skipped 1 previous similar message [ 2641.076670] Lustre: lustre-MDT0001: new disk, initializing [ 2641.133248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2641.157624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2641.164644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2643.142823] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2648.156245] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2650.891760] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2654.271073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2654.442631] Lustre: lustre-OST0000: new disk, initializing [ 2654.446325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2654.451463] Lustre: 161485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2654.506929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2655.828876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2655.835662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2655.894641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2657.295365] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2662.318384] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2664.674793] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2665.596116] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:52:06 (1782859926) [ 2768.864231] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.864542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2768.871747] Lustre: Skipped 2 previous similar messages [ 2768.875409] Lustre: Skipped 1 previous similar message [ 2773.984468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2773.988871] Lustre: Skipped 2 previous similar messages [ 2774.330366] Lustre: server umount lustre-MDT0000 complete [ 2776.034284] LustreError: 160313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782860038 with bad export cookie 18334622062449208764 [ 2776.036097] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2776.041990] LustreError: 160313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2779.103638] LustreError: 160014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.103774] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2779.113337] LustreError: 160014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2779.120900] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2781.151702] LustreError: 159526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.151719] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2786.271559] LustreError: 159526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.271564] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2790.367116] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2790.464196] Lustre: server umount lustre-MDT0001 complete [ 2798.438788] Lustre: server umount lustre-OST0000 complete [ 2802.539279] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 2806.966979] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2823.889400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.021410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2824.039881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2824.089085] Lustre: lustre-MDT0000: new disk, initializing [ 2824.117711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.128154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2825.854412] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2831.321805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2831.390363] Lustre: 166118:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2831.431502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2831.435501] Lustre: Skipped 1 previous similar message [ 2831.483616] Lustre: lustre-MDT0001: new disk, initializing [ 2831.538978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2831.543867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2833.326409] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2838.186296] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2840.844563] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2844.129801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2844.288356] Lustre: lustre-OST0000: new disk, initializing [ 2844.291584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2844.296327] Lustre: 168088:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2844.339217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2844.344655] Lustre: Skipped 1 previous similar message [ 2845.965708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2845.976318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2846.032706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2847.101351] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2851.890160] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2853.760994] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2853.880455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2855.810974] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2855.931469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2861.540654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2861.541725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.545708] LustreError: Skipped 1 previous similar message [ 2861.554664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2861.560726] Lustre: Skipped 1 previous similar message [ 2871.263179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2871.339688] Lustre: server umount lustre-OST0000 complete [ 2876.898300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.906568] Lustre: Skipped 2 previous similar messages [ 2879.349634] Lustre: server umount lustre-MDT0000 complete [ 2881.131699] LustreError: 166111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782860143 with bad export cookie 18334622062449367594 [ 2881.136739] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2881.139526] LustreError: 166111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2881.315826] Lustre: server umount lustre-MDT0001 complete [ 2884.876221] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:55:46 (1782860146) [ 2892.363236] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 2896.744050] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2914.174160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.317060] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2914.365562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2914.422992] Lustre: lustre-MDT0000: new disk, initializing [ 2914.455433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2914.465719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2916.497750] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2922.136412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.210312] Lustre: 172837:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2922.364991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2924.167671] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2929.203550] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2931.862970] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2935.174977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2935.369607] Lustre: 174814:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2935.761900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2935.829665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2938.193731] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2943.187203] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2946.021437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2946.025481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.026462] LustreError: Skipped 1 previous similar message [ 2946.036300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2946.041898] Lustre: Skipped 5 previous similar messages [ 2952.045614] Lustre: server umount lustre-OST0000 complete [ 2956.259141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.267052] Lustre: Skipped 2 previous similar messages [ 2960.069600] Lustre: server umount lustre-MDT0000 complete [ 2961.376967] LustreError: 172848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.388459] LustreError: 172848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2961.895657] LustreError: 172830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782860224 with bad export cookie 18334622062449368476 [ 2961.901734] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2961.903053] LustreError: 172830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2962.085759] Lustre: server umount lustre-MDT0001 complete [ 2966.235380] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:57:07 (1782860227) [ 2973.978564] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 2978.549866] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2995.591144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2995.740831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2995.759869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2995.763679] Lustre: Skipped 3 previous similar messages [ 2995.812427] Lustre: lustre-MDT0000: new disk, initializing [ 2995.814216] Lustre: Skipped 2 previous similar messages [ 2995.848773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2995.851883] Lustre: Skipped 2 previous similar messages [ 2995.859158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2995.864624] Lustre: Skipped 2 previous similar messages [ 2997.857127] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3003.449962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3003.530178] Lustre: 179128:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3003.690418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3005.606319] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3010.531673] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3013.118713] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3016.368683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3016.546061] Lustre: 181108:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3019.285607] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3024.177585] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3026.005742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3036.135807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3036.136548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.139926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3036.139936] Lustre: Skipped 4 previous similar messages [ 3036.140872] LustreError: Skipped 1 previous similar message [ 3036.148769] Lustre: Skipped 1 previous similar message [ 3046.367208] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3046.466496] Lustre: server umount lustre-OST0000 complete [ 3049.952564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3054.465533] Lustre: server umount lustre-MDT0000 complete [ 3056.267139] LustreError: 179929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782860318 with bad export cookie 18334622062449369561 [ 3056.273809] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3056.275472] LustreError: 179929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3056.440177] Lustre: server umount lustre-MDT0001 complete [ 3061.198966] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3062.824779] Key type lgssc unregistered [ 3063.002634] LNet: 182963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3063.007650] LNetError: 182963:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3063.016721] LNet: Removed LNI 192.168.204.113@tcp [ 3063.513348] Key type .llcrypt unregistered [ 3063.515977] Key type ._llcrypt unregistered [ 3075.528584] Key type ._llcrypt registered [ 3075.530843] Key type .llcrypt registered [ 3075.603826] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3075.971600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3075.993292] alg: No test for adler32 (adler32-zlib) [ 3076.923148] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 3077.050638] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3078.647159] Key type lgssc registered [ 3079.258527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3083.139655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3083.151022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.365343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3086.176089] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3090.233249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3090.434949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3092.440916] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3095.900848] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3098.451964] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3101.685349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3101.868809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3104.509986] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3107.898983] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3109.043163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3130.853402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.853573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3130.863674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.871159] Lustre: Skipped 1 previous similar message [ 3135.972448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3135.976906] Lustre: Skipped 1 previous similar message [ 3136.360474] Lustre: server umount lustre-OST0000 complete [ 3141.091844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3141.093309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3141.099985] Lustre: Skipped 2 previous similar messages [ 3144.311344] Lustre: server umount lustre-MDT0000 complete [ 3146.131070] LustreError: 186198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782860408 with bad export cookie 9020834198190468063 [ 3146.139147] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3146.139244] LustreError: 186198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3146.322491] Lustre: server umount lustre-MDT0001 complete [ 3151.213910] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3152.818097] Key type lgssc unregistered [ 3153.003593] LNet: 187909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3153.009301] LNetError: 187909:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3153.018729] LNet: Removed LNI 192.168.204.113@tcp [ 3153.530437] Key type .llcrypt unregistered [ 3153.532856] Key type ._llcrypt unregistered [ 3165.142753] Key type ._llcrypt registered [ 3165.144626] Key type .llcrypt registered [ 3165.211645] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3165.673454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3165.750304] alg: No test for adler32 (adler32-zlib) [ 3166.643944] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 3166.792832] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3168.407160] Key type lgssc registered [ 3169.037338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3172.876790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3172.888326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.124102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.066590] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3180.190998] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.399983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3182.364982] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3185.846916] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3188.437618] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3191.733605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3191.934026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3194.763181] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3198.295475] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3199.153123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3211.750665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3211.752384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3211.757373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3211.764120] Lustre: Skipped 1 previous similar message [ 3211.770729] Lustre: Skipped 1 previous similar message [ 3213.032184] Lustre: server umount lustre-OST0000 complete [ 3216.407426] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9c8c86f8c800.ost_server_uuid 50 [ 3231.849268] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3242.465401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3242.466294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3242.473643] Lustre: Skipped 1 previous similar message [ 3247.584554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.587362] Lustre: Skipped 2 previous similar messages [ 3248.306274] Lustre: server umount lustre-MDT0000 complete [ 3250.036854] LustreError: 189284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782860512 with bad export cookie 13095385379785353926 [ 3250.044808] LustreError: 189284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3250.045226] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3250.200981] Lustre: server umount lustre-MDT0001 complete [ 3254.056799] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:01:55 (1782860515) [ 3257.123397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.343062] LustreError: 193255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.349875] LustreError: 193255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3257.370623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3259.175817] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3262.433056] LustreError: 193256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3263.115447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.285590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3265.016745] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3268.307842] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3270.794391] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3273.947917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3274.129291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3276.723165] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3280.024633] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3282.347939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3283.219200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3283.356651] Lustre: lustre-OST0001: new disk, initializing [ 3283.359925] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3283.365121] Lustre: 196094:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3286.062189] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3290.916867] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3292.187804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3292.195122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3292.195836] Lustre: Skipped 1 previous similar message [ 3292.202327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3292.208583] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3292.249694] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3296.236443] Lustre: lustre-OST0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 3296.241035] Lustre: Skipped 1 previous similar message [ 3298.312377] Lustre: server umount lustre-OST0000 complete [ 3301.132879] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3301.263594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3301.343418] LustreError: 195078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.234316] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3303.364623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3306.462996] LustreError: 195076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3306.473812] LustreError: 195076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3311.584132] LustreError: 195979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3311.595706] LustreError: 195979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3317.730901] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3317.735347] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3317.735709] LustreError: Skipped 1 previous similar message [ 3317.744951] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3317.750469] Lustre: Skipped 3 previous similar messages [ 3319.861465] Lustre: server umount lustre-OST0001 complete [ 3322.850260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3322.858932] Lustre: Skipped 2 previous similar messages [ 3327.875742] Lustre: server umount lustre-MDT0000 complete [ 3327.969937] LustreError: 193256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3327.979113] LustreError: 193256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3329.677178] LustreError: 193236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782860592 with bad export cookie 13095385379785354983 [ 3329.684946] LustreError: 193236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3329.686387] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3329.836379] Lustre: server umount lustre-MDT0001 complete [ 3341.145811] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:03:22 (1782860602) [ 3344.390693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3344.520926] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3344.543008] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3344.644451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3344.646446] Lustre: Skipped 1 previous similar message [ 3346.544269] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3350.643870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.696945] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3350.716813] Lustre: 199510:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3352.791770] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3356.229175] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3358.811192] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3362.051681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3362.148468] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3364.994557] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3368.372753] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3368.505493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3371.572081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3371.619914] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3374.499390] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3377.943268] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3383.276725] Lustre: lustre-OST0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 3383.281230] Lustre: Skipped 2 previous similar messages [ 3383.778405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3383.779984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.783540] LustreError: Skipped 1 previous similar message [ 3385.356152] Lustre: server umount lustre-OST0000 complete [ 3388.381618] LustreError: 202042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3399.659036] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3399.664070] LustreError: Skipped 1 previous similar message [ 3402.668560] Lustre: server umount lustre-OST0001 complete [ 3404.770798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3404.778637] Lustre: Skipped 4 previous similar messages [ 3410.692270] Lustre: server umount lustre-MDT0000 complete [ 3412.496191] LustreError: 199502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782860674 with bad export cookie 13095385379785356369 [ 3412.503639] LustreError: 199502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3412.504450] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3412.663966] Lustre: server umount lustre-MDT0001 complete [ 3423.823456] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:04:45 (1782860685) [ 3431.362648] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 3435.887541] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3452.366756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3452.497572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3452.515866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3452.572430] Lustre: lustre-MDT0000: new disk, initializing [ 3452.602512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3452.606788] Lustre: Skipped 3 previous similar messages [ 3452.616232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3454.361389] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3459.716618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3459.781330] Lustre: 207576:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3459.837501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3459.841506] Lustre: Skipped 1 previous similar message [ 3459.894226] Lustre: lustre-MDT0001: new disk, initializing [ 3459.959949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3459.966220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3461.674311] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3466.394890] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3468.961594] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3472.087981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3472.244465] Lustre: lustre-OST0000: new disk, initializing [ 3472.247344] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3472.251483] Lustre: 209546:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3474.865181] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3479.594671] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3481.453785] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3482.120497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3482.127207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3482.178037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3482.606363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3484.361974] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3484.441807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3487.203709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3487.206712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3487.209574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3487.216593] Lustre: Skipped 1 previous similar message [ 3487.220677] Lustre: Skipped 10 previous similar messages [ 3491.431839] Lustre: server umount lustre-OST0000 complete [ 3495.904468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3495.908849] LustreError: Skipped 1 previous similar message [ 3499.307844] Lustre: server umount lustre-MDT0000 complete [ 3501.016066] LustreError: 207570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782860763 with bad export cookie 13095385379785357902 [ 3501.022405] LustreError: 207570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3501.022993] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3501.165945] Lustre: server umount lustre-MDT0001 complete [ 3505.035046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.233034] LustreError: 211768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3505.240276] LustreError: 211768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3506.912620] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3510.839807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3512.701405] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3516.055131] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3518.565161] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3521.687284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3521.837007] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3521.840204] Lustre: Skipped 4 previous similar messages [ 3524.292503] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3527.549590] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3529.528838] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3530.679249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3532.504982] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3536.730745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 3540.447966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3543.756406] Lustre: server umount lustre-OST0000 complete [ 3545.759461] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3545.834137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3547.275628] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3571.756993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3574.435354] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3577.506186] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3578.248320] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3580.770356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3580.773970] Lustre: Skipped 7 previous similar messages [ 3585.869291] Lustre: server umount lustre-OST0000 complete [ 3587.114042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.118073] Lustre: Skipped 4 previous similar messages [ 3601.375140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3601.438179] Lustre: server umount lustre-MDT0000 complete [ 3602.690293] LustreError: 211749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782860865 with bad export cookie 13095385379785358763 [ 3602.693102] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3602.693711] LustreError: 211749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3602.805661] Lustre: server umount lustre-MDT0001 complete [ 3605.656504] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:07:47 (1782860867) [ 3607.928026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.083050] LustreError: 217779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3608.090165] LustreError: 217779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3609.465092] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3612.529274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.156684] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3616.732181] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3618.503038] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3620.675862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3622.610983] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3624.916146] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3626.214295] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3633.421718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3634.773430] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3634.842842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3637.095163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3637.163295] Lustre: lustre-OST0001: new disk, initializing [ 3637.164966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3637.167483] Lustre: 221195:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3639.145344] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3642.932744] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3643.392897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3643.398178] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3643.430108] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3644.359469] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3644.430476] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3645.810374] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3645.886290] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3648.482455] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3648.485256] LustreError: Skipped 2 previous similar messages [ 3652.670175] Lustre: server umount lustre-OST0001 complete [ 3654.494119] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3654.557591] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3655.814537] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3655.883590] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3679.383283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3679.445944] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3679.449473] Lustre: Skipped 5 previous similar messages [ 3681.271616] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3683.535159] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3684.073181] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3690.807334] Lustre: server umount lustre-OST0001 complete [ 3710.434436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3710.436309] Lustre: Skipped 22 previous similar messages [ 3717.087169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3717.167751] Lustre: server umount lustre-OST0000 complete [ 3720.160465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3720.165767] LustreError: Skipped 2 previous similar messages [ 3720.168857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3720.176941] Lustre: Skipped 5 previous similar messages [ 3725.257456] Lustre: server umount lustre-MDT0000 complete [ 3726.767618] LustreError: 217807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782860989 with bad export cookie 13095385379785359708 [ 3726.772087] LustreError: 217807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3726.772647] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3741.151124] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3741.248901] Lustre: server umount lustre-MDT0001 complete [ 3749.724556] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:10:11 (1782861011) [ 3752.443694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.558052] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3752.573269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3754.368749] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3758.316077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.373403] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3758.387510] Lustre: 226427:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3758.443628] LustreError: 226439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3758.453954] LustreError: 226439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 29 previous similar messages [ 3760.016644] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3763.123827] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3765.669914] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3768.819832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3768.908183] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3771.396318] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3774.630260] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3779.128913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3779.158316] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3781.266982] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3784.158533] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3784.821551] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3786.295682] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3786.366069] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3788.014583] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3788.134367] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3790.134072] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c8c86e32000.ost_server_uuid 50 [ 3790.689411] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c8c86e32000.ost_server_uuid in FULL state after 0 sec [ 3791.090186] Lustre: Permanently deactivating lustre-OST0001 [ 3791.094823] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3791.998472] Lustre: Permanently reactivating lustre-OST0001 [ 3799.164707] Lustre: server umount lustre-OST0001 complete [ 3806.411388] Lustre: server umount lustre-OST0000 complete [ 3813.719351] Lustre: server umount lustre-MDT0000 complete [ 3814.857124] LustreError: 226421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782861077 with bad export cookie 13095385379785360968 [ 3814.860480] LustreError: 226421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3814.860498] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3814.968050] Lustre: server umount lustre-MDT0001 complete [ 3822.509381] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:11:24 (1782861084) [ 3824.784643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3824.870573] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3824.879478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3824.882165] Lustre: Skipped 5 previous similar messages [ 3826.160244] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3828.767417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3828.796698] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3828.805184] Lustre: 233110:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3830.130902] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3832.334690] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3833.978238] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3836.069919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3836.146845] Lustre: Permanently deactivating lustre-OST0000 [ 3836.150637] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3837.942134] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3839.923543] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3840.156462] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3842.247972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3844.128893] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3846.364725] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3847.727938] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3852.141766] Lustre: Permanently reactivating lustre-OST0000 [ 3855.241462] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3855.246063] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3855.251529] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3855.273295] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3855.280149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3857.895179] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3857.897222] LustreError: Skipped 4 previous similar messages [ 3861.954804] Lustre: server umount lustre-OST0001 complete [ 3869.258745] Lustre: server umount lustre-OST0000 complete [ 3871.027730] Lustre: server umount lustre-MDT0000 complete [ 3872.771153] LustreError: 233102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782861135 with bad export cookie 13095385379785362620 [ 3872.778955] LustreError: 233102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3872.778981] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3878.988456] Lustre: server umount lustre-MDT0001 complete [ 3882.538365] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3883.663713] Key type lgssc unregistered [ 3883.831737] LNet: 237962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3883.836482] LNetError: 237962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3883.849478] LNet: Removed LNI 192.168.204.113@tcp [ 3884.263755] Key type .llcrypt unregistered [ 3884.265671] Key type ._llcrypt unregistered [ 3892.382133] Key type ._llcrypt registered [ 3892.383220] Key type .llcrypt registered [ 3892.414520] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3892.694881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3892.722116] alg: No test for adler32 (adler32-zlib) [ 3893.566307] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 3893.644196] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3895.215089] Key type lgssc registered [ 3895.530428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3898.435528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3898.441363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3899.573726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3901.247109] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3905.163086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3905.227277] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3905.244630] Lustre: 239392:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3905.253124] Lustre: Permanently deactivating lustre-MDT0001 [ 3905.262702] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3905.438846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3905.919068] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3905.921425] LustreError: 239435:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3905.924864] LustreError: 239435:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3905.930337] LustreError: 239435:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 3907.248638] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3910.578888] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3912.733714] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3915.925397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3916.117052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.879360] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3921.874133] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3924.324458] LustreError: 241245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3924.330671] LustreError: 241245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3924.331902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3925.120161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3925.241893] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3928.114223] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3930.354733] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3930.357307] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3931.010752] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3932.907975] Lustre: Permanently reactivating lustre-MDT0001 [ 3932.919413] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3932.923850] Lustre: Skipped 2 previous similar messages [ 3940.854802] Lustre: lustre-MDT0001: Client 90df5373-ae45-4992-bd7a-ddf6344a5b37 (at 192.168.204.13@tcp) reconnecting [ 3943.869709] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3944.796894] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3947.143597] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3947.230682] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3948.873521] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3948.976859] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3950.209173] Lustre: Permanently deactivating lustre-MDT0001 [ 3950.219751] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3950.224569] Lustre: Skipped 2 previous similar messages [ 3959.316762] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3966.435547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.437584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3966.440173] Lustre: Skipped 3 previous similar messages [ 3966.442161] Lustre: Skipped 2 previous similar messages [ 3970.092585] Lustre: server umount lustre-MDT0000 complete [ 3971.553728] LustreError: 239404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3971.559182] LustreError: 239404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3971.566956] LustreError: 240143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782861233 with bad export cookie 1438956701231680708 [ 3971.570491] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3971.571730] LustreError: 240143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3971.672976] Lustre: server umount lustre-MDT0001 complete [ 3979.398422] Lustre: server umount lustre-OST0000 complete [ 3984.555899] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 19:14:05 (1782861245) [ 3989.298711] Lustre: server umount lustre-OST0001 complete [ 3991.962960] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 3996.156525] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4012.510814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.674817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4012.681050] Lustre: Skipped 2 previous similar messages [ 4012.693904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4012.752709] Lustre: lustre-MDT0000: new disk, initializing [ 4012.784900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4012.795996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4014.552652] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4020.146183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4020.206513] Lustre: 247746:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4020.239639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4020.242020] Lustre: Skipped 1 previous similar message [ 4020.277572] Lustre: lustre-MDT0001: new disk, initializing [ 4020.316584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4020.319762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4021.734458] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4026.221151] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4028.851367] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4032.129877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4032.225023] Lustre: lustre-OST0000: new disk, initializing [ 4032.226879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4032.229541] Lustre: 249716:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4032.256682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4032.258882] Lustre: Skipped 1 previous similar message [ 4034.188431] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4034.626076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4034.630303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4034.664367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4038.715613] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4039.737851] LustreError: 247755:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4044.902490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4045.013866] Lustre: lustre-OST0001: new disk, initializing [ 4045.017428] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4045.022385] Lustre: 250829:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4047.654071] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4049.831163] LustreError: 247755:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4049.850562] LustreError: 247754:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4052.468647] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4053.304407] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4054.553778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4054.559260] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4054.613355] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4059.951106] LustreError: 247754:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4062.190456] Lustre: lustre-OST0001: Not available for connect from 192.168.204.13@tcp (stopping) [ 4064.742562] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4064.743131] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4064.744192] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4064.746703] LustreError: Skipped 1 previous similar message [ 4064.753100] Lustre: Skipped 1 previous similar message [ 4066.944841] Lustre: server umount lustre-OST0001 complete [ 4067.295400] LustreError: 249700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.864971] LustreError: 249701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.875747] LustreError: 249701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4074.983377] LustreError: 249702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.994063] LustreError: 249702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4080.099419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.104410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4080.107202] Lustre: Skipped 1 previous similar message [ 4080.111752] Lustre: Skipped 1 previous similar message [ 4085.219785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4085.221304] LustreError: 249702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4085.223668] Lustre: Skipped 1 previous similar message [ 4085.234721] LustreError: 249702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4085.735548] Lustre: server umount lustre-OST0000 complete [ 4090.336941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4090.341077] Lustre: Skipped 1 previous similar message [ 4093.207545] Lustre: server umount lustre-MDT0000 complete [ 4094.280129] LustreError: 247739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782861356 with bad export cookie 1438956701231683298 [ 4094.283732] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4094.284100] LustreError: 247739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4100.474818] Lustre: server umount lustre-MDT0001 complete [ 4105.179199] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4106.544278] Key type lgssc unregistered [ 4106.693417] LNet: 252859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4106.696156] LNetError: 252859:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4106.702355] LNet: Removed LNI 192.168.204.113@tcp [ 4106.986765] Key type .llcrypt unregistered [ 4106.987806] Key type ._llcrypt unregistered [ 4126.147713] Key type ._llcrypt registered [ 4126.149184] Key type .llcrypt registered [ 4126.195616] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4126.516596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4126.582888] alg: No test for adler32 (adler32-zlib) [ 4127.470589] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 4127.575738] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4129.167157] Key type lgssc registered [ 4129.756416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4133.407802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4133.419300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.538427] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4134.549916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4134.622718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4136.470593] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4140.441087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4140.486950] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4140.501407] Lustre: 255043:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4140.644969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4142.423522] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4145.570197] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4147.950005] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4151.278577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4151.375814] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4151.449274] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4151.516113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4154.243978] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4156.913494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4157.678293] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4197.859116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4197.860515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.864173] LustreError: Skipped 1 previous similar message [ 4197.873562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4197.879161] Lustre: Skipped 1 previous similar message [ 4202.980087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4202.984084] Lustre: Skipped 1 previous similar message [ 4203.517067] Lustre: server umount lustre-OST0000 complete [ 4205.233149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4210.972594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4211.122652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4213.431748] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4216.965802] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4228.521679] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4228.529352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4228.529510] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4228.542491] Lustre: Skipped 1 previous similar message [ 4254.180681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4254.182213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4254.183551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4254.188823] Lustre: Skipped 2 previous similar messages [ 4254.194198] LustreError: Skipped 1 previous similar message [ 4257.141213] Lustre: server umount lustre-OST0000 complete [ 4259.298202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.307594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4259.311891] Lustre: Skipped 2 previous similar messages [ 4265.107950] Lustre: server umount lustre-MDT0000 complete [ 4266.726957] LustreError: 256904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782861529 with bad export cookie 8050324488447093551 [ 4266.732448] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4266.734540] LustreError: 256904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4266.887450] Lustre: server umount lustre-MDT0001 complete [ 4271.855918] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4273.120620] Key type lgssc unregistered [ 4273.268821] LNet: 260462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4273.273985] LNetError: 260462:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4273.285579] LNet: Removed LNI 192.168.204.113@tcp [ 4273.756527] Key type .llcrypt unregistered [ 4273.758962] Key type ._llcrypt unregistered [ 4285.870436] Key type ._llcrypt registered [ 4285.871715] Key type .llcrypt registered [ 4285.912874] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4286.213658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4286.240073] alg: No test for adler32 (adler32-zlib) [ 4287.130640] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 4287.255533] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4288.855192] Key type lgssc registered [ 4289.447668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4293.070356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4293.082101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.287817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4296.007711] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4300.107626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4300.286419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4302.134576] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4305.749738] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4308.427397] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4311.636608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4311.808697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4314.320507] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4317.785336] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4319.020121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4339.680553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4339.685787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.694843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4340.705699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4340.708128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4340.714940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4343.973906] Lustre: server umount lustre-OST0000 complete [ 4345.824669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.825620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4345.831293] Lustre: Skipped 1 previous similar message [ 4345.834264] Lustre: Skipped 1 previous similar message [ 4350.944851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4350.948850] Lustre: Skipped 2 previous similar messages [ 4356.064804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4356.068376] Lustre: Skipped 1 previous similar message [ 4360.159131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4360.247163] Lustre: server umount lustre-MDT0000 complete [ 4361.184607] LustreError: 261855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4361.191800] LustreError: 261855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4362.024907] LustreError: 261837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782861624 with bad export cookie 11129504204392794609 [ 4362.030212] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4362.032571] LustreError: 261837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4362.190524] Lustre: server umount lustre-MDT0001 complete [ 4367.195239] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4368.616917] Key type lgssc unregistered [ 4368.785572] LNet: 266183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4368.790578] LNetError: 266183:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4368.799517] LNet: Removed LNI 192.168.204.113@tcp [ 4369.222459] Key type .llcrypt unregistered [ 4369.224777] Key type ._llcrypt unregistered [ 4378.453172] Key type ._llcrypt registered [ 4378.454816] Key type .llcrypt registered [ 4378.518984] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4378.849174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4378.890268] alg: No test for adler32 (adler32-zlib) [ 4379.752066] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 4379.836642] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4381.415165] Key type lgssc registered [ 4381.985820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4385.635485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4385.645510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4386.797944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4388.169215] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4392.047285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4392.156578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4393.855772] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4397.095348] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4399.702322] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4402.895897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4403.074985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4405.797265] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4409.229148] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4412.334142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4426.722530] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4426.727651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4426.736446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4427.748944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4427.749825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4427.757194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4430.173639] Lustre: server umount lustre-OST0000 complete [ 4432.867066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.868053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4432.875143] Lustre: Skipped 1 previous similar message [ 4432.879149] Lustre: Skipped 1 previous similar message [ 4437.985198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4437.989725] Lustre: Skipped 2 previous similar messages [ 4438.125774] Lustre: server umount lustre-MDT0000 complete [ 4439.806168] LustreError: 267362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782861702 with bad export cookie 7725903336159676422 [ 4439.811500] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4439.812846] LustreError: 267362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4439.953449] Lustre: server umount lustre-MDT0001 complete [ 4443.893724] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4445.155160] Key type lgssc unregistered [ 4445.288408] LNet: 271079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4445.292286] LNetError: 271079:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4445.301311] LNet: Removed LNI 192.168.204.113@tcp [ 4445.602218] Key type .llcrypt unregistered [ 4445.603863] Key type ._llcrypt unregistered [ 4456.845763] Key type ._llcrypt registered [ 4456.847661] Key type .llcrypt registered [ 4456.909176] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4457.213566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4457.221764] alg: No test for adler32 (adler32-zlib) [ 4458.080198] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 4458.186046] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4459.775178] Key type lgssc registered [ 4460.311654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4464.021654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4464.032944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4465.194555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4467.075922] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4471.191231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.359326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4473.178670] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4476.630970] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4479.337411] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4482.530091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4482.670571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4485.199919] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4488.708038] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4489.835475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4510.688193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4510.692970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4510.703196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4512.737279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4512.741170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4512.742581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.537158] Lustre: server umount lustre-OST0000 complete [ 4517.344513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4517.349664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.358486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4517.362682] Lustre: Skipped 1 previous similar message [ 4522.479550] Lustre: server umount lustre-MDT0000 complete [ 4522.977909] LustreError: 272472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4522.988160] LustreError: 272472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4524.251772] LustreError: 272497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782861786 with bad export cookie 12388444246024420626 [ 4524.258557] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4524.259469] LustreError: 272497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4524.417978] Lustre: server umount lustre-MDT0001 complete [ 4529.554612] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4530.917299] Key type lgssc unregistered [ 4531.081711] LNet: 276803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4531.086835] LNetError: 276803:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4531.098496] LNet: Removed LNI 192.168.204.113@tcp [ 4531.540808] Key type .llcrypt unregistered [ 4531.543189] Key type ._llcrypt unregistered [ 4541.057591] Key type ._llcrypt registered [ 4541.059427] Key type .llcrypt registered [ 4541.119382] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4541.461824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4541.503184] alg: No test for adler32 (adler32-zlib) [ 4542.374928] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 4542.468631] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4544.063135] Key type lgssc registered [ 4544.659239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4548.519229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4548.530791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4549.737931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4551.581505] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4555.723425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4555.911135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4557.618494] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4561.178413] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4563.909579] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4567.118729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4567.283518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.942446] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4573.270851] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4575.470700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4591.073407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4591.075300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4591.078459] LustreError: Skipped 1 previous similar message [ 4591.091298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4592.100283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4592.102299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4592.105929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4594.158271] Lustre: server umount lustre-OST0000 complete [ 4597.218208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4597.219444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4597.220756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4597.220767] Lustre: Skipped 1 previous similar message [ 4597.226302] Lustre: Skipped 1 previous similar message [ 4598.190359] Lustre: server umount lustre-MDT0000 complete [ 4600.039998] LustreError: 277984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782861862 with bad export cookie 18114017074048895398 [ 4600.045848] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4600.048659] LustreError: 277984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4602.262744] Lustre: server umount lustre-MDT0001 complete [ 4607.392501] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4608.910774] Key type lgssc unregistered [ 4609.078682] LNet: 281735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4609.083834] LNetError: 281735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4609.096535] LNet: Removed LNI 192.168.204.113@tcp [ 4609.553542] Key type .llcrypt unregistered [ 4609.556179] Key type ._llcrypt unregistered [ 4614.644248] Key type ._llcrypt registered [ 4614.645484] Key type .llcrypt registered [ 4614.689129] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4622.800688] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4630.595024] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4630.960847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4630.993625] alg: No test for adler32 (adler32-zlib) [ 4631.851997] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 4631.940147] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4633.535163] Key type lgssc registered [ 4634.143755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4651.514310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4651.525900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4652.640836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4652.656894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4652.702709] Lustre: lustre-MDT0000: new disk, initializing [ 4652.765060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4652.777701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4654.583701] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4660.161790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4660.235517] Lustre: 285258:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4660.278145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4660.282100] Lustre: Skipped 1 previous similar message [ 4660.334227] Lustre: lustre-MDT0001: new disk, initializing [ 4660.383927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4660.404426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4660.411262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4662.297077] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4667.348750] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4670.003783] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4673.274399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4673.446114] Lustre: lustre-OST0000: new disk, initializing [ 4673.449928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4673.455679] Lustre: 287226:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4673.509143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.159567] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4679.178876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4679.184128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4679.234268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4680.963023] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4682.721808] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4682.790300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4684.563414] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4684.680105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4689.376480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4689.378352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.379803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4689.382421] Lustre: Skipped 1 previous similar message [ 4691.407540] Lustre: server umount lustre-OST0000 complete [ 4694.496562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4694.497260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4694.500241] Lustre: Skipped 1 previous similar message [ 4694.502034] Lustre: Skipped 1 previous similar message [ 4698.706650] Lustre: server umount lustre-MDT0000 complete [ 4699.833070] LustreError: 285251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782861962 with bad export cookie 186301919557105121 [ 4699.835554] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4699.836604] LustreError: 285251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4699.944884] Lustre: server umount lustre-MDT0001 complete [ 4702.328337] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:26:03 (1782861963) [ 4704.333760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4704.507309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4705.648195] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4708.198416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4709.457155] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4711.643620] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4713.249786] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4715.263034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4715.369309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4715.371530] Lustre: Skipped 1 previous similar message [ 4717.091367] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4719.994898] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4733.929503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4733.931262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.934604] LustreError: Skipped 1 previous similar message [ 4733.943827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4733.949135] Lustre: Skipped 2 previous similar messages [ 4739.050260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4739.055097] Lustre: Skipped 1 previous similar message [ 4740.073116] Lustre: server umount lustre-OST0000 complete [ 4744.162462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4744.162552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4744.164129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4744.170788] Lustre: Skipped 2 previous similar messages [ 4748.000878] Lustre: server umount lustre-MDT0000 complete [ 4749.280979] LustreError: 289635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4749.291410] LustreError: 289635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4749.724926] LustreError: 289616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782862012 with bad export cookie 186301919557105989 [ 4749.730833] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4749.732510] LustreError: 289616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4749.874599] Lustre: server umount lustre-MDT0001 complete [ 4754.772473] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4755.665085] Key type lgssc unregistered [ 4755.781432] LNet: 293190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4755.783933] LNetError: 293190:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4755.792278] LNet: Removed LNI 192.168.204.113@tcp [ 4756.067315] Key type .llcrypt unregistered [ 4756.068910] Key type ._llcrypt unregistered [ 4765.260219] Key type ._llcrypt registered [ 4765.261336] Key type .llcrypt registered [ 4765.293315] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4772.416992] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4772.693808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4772.739062] alg: No test for adler32 (adler32-zlib) [ 4773.627414] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 4773.720779] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4775.303080] Key type lgssc registered [ 4775.610533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4791.321562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4791.326732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4792.402483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4792.411706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4792.446322] Lustre: lustre-MDT0000: new disk, initializing [ 4792.466416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4792.471949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4793.700573] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4798.226562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4798.302635] Lustre: 296986:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4798.346513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4798.350379] Lustre: Skipped 1 previous similar message [ 4798.400696] Lustre: lustre-MDT0001: new disk, initializing [ 4798.445718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4798.466677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4798.473469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4800.157444] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4805.132579] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4807.666937] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4810.964063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4811.111832] Lustre: lustre-OST2710: new disk, initializing [ 4811.115441] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4811.120611] Lustre: 298954:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4811.170640] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4812.935685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4812.942910] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4812.998136] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4813.825091] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4818.519660] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4821.905114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4822.011604] Lustre: lustre-OST03e8: new disk, initializing [ 4822.014748] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4822.019329] Lustre: 300016:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4822.062737] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4824.753463] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4828.364415] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4829.182265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4829.186894] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4829.210257] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4831.657663] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4831.777821] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4833.334811] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4833.454730] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4835.153408] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4835.258740] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4837.079249] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4837.145205] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4839.393210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4839.395493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4839.397092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4839.399220] Lustre: Skipped 2 previous similar messages [ 4839.401076] Lustre: Skipped 1 previous similar message [ 4844.502111] Lustre: server umount lustre-MDT0000 complete [ 4845.720964] LustreError: 296979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782862108 with bad export cookie 5610159165024461350 [ 4845.726863] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4845.728635] LustreError: 296979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4845.862192] Lustre: server umount lustre-MDT0001 complete [ 4857.878820] Lustre: server umount lustre-OST2710 complete [ 4869.626350] Lustre: server umount lustre-OST03e8 complete [ 4878.086993] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4882.215723] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4898.736375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4898.859575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4898.876562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4898.927117] Lustre: lustre-MDT0000: new disk, initializing [ 4898.957195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4898.966663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4900.887773] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4906.325696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4906.384417] Lustre: 305258:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4906.513101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4908.241161] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4913.263383] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4915.936383] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4919.105524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4919.245275] Lustre: lustre-OST0000: new disk, initializing [ 4919.246863] Lustre: Skipped 1 previous similar message [ 4919.248851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4919.251467] Lustre: Skipped 2 previous similar messages [ 4919.253685] Lustre: 307227:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4919.284417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4919.287378] Lustre: Skipped 1 previous similar message [ 4921.730314] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4924.932851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4924.938901] Lustre: Skipped 1 previous similar message [ 4924.942521] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4924.993750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4926.574439] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4928.445289] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4928.559326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4930.339493] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4930.431957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4935.141823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4935.142672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4935.146751] LustreError: Skipped 1 previous similar message [ 4935.147537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4935.155587] Lustre: Skipped 2 previous similar messages [ 4935.164404] Lustre: Skipped 1 previous similar message [ 4937.384190] Lustre: server umount lustre-OST0000 complete [ 4940.258295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.259443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4940.265919] Lustre: Skipped 1 previous similar message [ 4940.271733] Lustre: Skipped 1 previous similar message [ 4945.329259] Lustre: server umount lustre-MDT0000 complete [ 4945.376978] LustreError: 305270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4945.388536] LustreError: 305270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4947.161105] LustreError: 305252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782862209 with bad export cookie 5610159165024463030 [ 4947.166663] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4947.168650] LustreError: 305252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4947.319878] Lustre: server umount lustre-MDT0001 complete [ 4951.649176] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:30:12 (1782862212) [ 4952.416226] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4953.291537] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:30:14 (1782862214) [ 4954.748898] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4967.142569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4967.248297] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4967.256969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4967.323686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4969.023875] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4973.101821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4973.155561] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4973.168903] Lustre: 311077:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4975.099279] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4992.164737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4992.274818] Lustre: Denying initial registration attempt from nid 192.168.204.113@tcp, specified as failover [ 4992.280483] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4992.285983] LustreError: 312532:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4992.290306] LustreError: 312532:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4992.293983] LustreError: 312532:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4992.323552] Lustre: server umount lustre-OST0000 complete [ 4992.325374] LustreError: 312532:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4996.065801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.066796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4996.075041] Lustre: Skipped 1 previous similar message [ 5001.185456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5001.189782] Lustre: Skipped 2 previous similar messages [ 5001.710820] Lustre: server umount lustre-MDT0000 complete [ 5003.553063] LustreError: 311069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782862265 with bad export cookie 5610159165024463898 [ 5003.558579] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5003.560720] LustreError: 311069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5003.706825] Lustre: server umount lustre-MDT0001 complete [ 5008.785567] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5010.239498] Key type lgssc unregistered [ 5010.378719] LNet: 313650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5010.383620] LNetError: 313650:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5010.394570] LNet: Removed LNI 192.168.204.113@tcp [ 5010.813077] Key type .llcrypt unregistered [ 5010.814493] Key type ._llcrypt unregistered [ 5015.792918] Key type ._llcrypt registered [ 5015.794435] Key type .llcrypt registered [ 5015.850809] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5016.171258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5016.179649] alg: No test for adler32 (adler32-zlib) [ 5017.028611] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 5017.123774] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5018.719200] Key type lgssc registered [ 5019.258510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5047.357133] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5050.699083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5050.709941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5051.812827] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5051.821608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5051.893159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5053.445856] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5057.465773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.496341] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5057.506038] Lustre: 316387:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5057.599859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5059.352345] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5062.919786] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5065.409212] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5081.700097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5081.814722] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5081.928697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5084.429019] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5087.685697] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5088.668663] Lustre: server umount lustre-OST0000 complete [ 5093.344293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5093.348705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5093.356307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5094.369124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5094.370240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.628140] Lustre: server umount lustre-MDT0000 complete [ 5097.912479] LustreError: 317131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782862360 with bad export cookie 2521484468995837824 [ 5097.918108] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5097.922395] LustreError: 317131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5098.048869] Lustre: server umount lustre-MDT0001 complete [ 5102.023804] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5102.984960] Key type lgssc unregistered [ 5103.122665] LNet: 319959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5103.127524] LNetError: 319959:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5103.143549] LNet: Removed LNI 192.168.204.113@tcp [ 5103.533901] Key type .llcrypt unregistered [ 5103.535093] Key type ._llcrypt unregistered [ 5113.983084] Key type ._llcrypt registered [ 5113.985153] Key type .llcrypt registered [ 5114.047880] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5114.343323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5114.351091] alg: No test for adler32 (adler32-zlib) [ 5115.221835] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 5115.333964] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5116.943218] Key type lgssc registered [ 5117.493578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5121.020692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5121.025802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.168622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5123.848818] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5127.948403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5128.108835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5130.022958] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5133.432991] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5135.903903] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5140.770285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5140.881139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5142.792371] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5145.068647] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5158.880283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5158.883781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5158.889991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5161.957587] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 5163.489036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5163.492574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.755463] Lustre: server umount lustre-MDT0000 complete [ 5165.960503] LustreError: 321334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782862428 with bad export cookie 16387900308241349687 [ 5165.964702] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5165.964964] LustreError: 321334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5167.068843] LustreError: 321346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5167.078631] LustreError: 321346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5167.083605] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.13@tcp (stopping) [ 5167.086717] Lustre: Skipped 2 previous similar messages [ 5168.607805] LustreError: 321376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5168.608070] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5168.616436] Lustre: Skipped 1 previous similar message [ 5170.655808] LustreError: 321346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.187968] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.13@tcp (stopping) [ 5172.190879] Lustre: Skipped 2 previous similar messages [ 5175.775975] LustreError: 321346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.784759] LustreError: 321346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5180.383129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5180.469386] Lustre: server umount lustre-MDT0001 complete [ 5182.729511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5185.701418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5191.135834] LustreError: 320663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009c39307e x1869466699852672/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5191.265533] LustreError: 323181:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.270283] LustreError: 323181:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5191.289044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5191.308622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5192.521322] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5195.399726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5195.533261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5196.830898] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5199.224742] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5205.991831] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5209.056953] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 5210.902548] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5211.509576] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5216.225740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5216.228340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5216.233138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5216.235595] Lustre: Skipped 3 previous similar messages [ 5226.975266] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5227.070937] Lustre: server umount lustre-OST0000 complete [ 5231.586313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5231.586882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.598454] Lustre: Skipped 2 previous similar messages [ 5236.704956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5236.709013] Lustre: Skipped 7 previous similar messages [ 5243.359166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5243.439900] Lustre: server umount lustre-MDT0000 complete [ 5245.189971] LustreError: 321333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782862507 with bad export cookie 16387900308241356806 [ 5245.196085] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5245.197587] LustreError: 321333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5245.349646] Lustre: server umount lustre-MDT0001 complete [ 5250.148485] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5251.521221] Key type lgssc unregistered [ 5251.692516] LNet: 327633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5251.697951] LNetError: 327633:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5251.709684] LNet: Removed LNI 192.168.204.113@tcp [ 5252.188979] Key type .llcrypt unregistered [ 5252.191271] Key type ._llcrypt unregistered [ 5264.311449] Key type ._llcrypt registered [ 5264.313145] Key type .llcrypt registered [ 5264.371677] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5264.699743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5264.754087] alg: No test for adler32 (adler32-zlib) [ 5265.620388] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 5265.717459] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5267.303182] Key type lgssc registered [ 5267.838550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5271.637583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5271.649051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5272.834233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5274.540922] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5278.571587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5278.724990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5280.433462] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5283.897720] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5286.588188] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5288.927821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5288.931367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5288.939183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5289.953529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5289.954759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5295.072942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5295.077556] Lustre: Skipped 2 previous similar messages [ 5300.192847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5300.197469] Lustre: Skipped 1 previous similar message [ 5302.239132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5302.317906] Lustre: server umount lustre-MDT0000 complete [ 5304.037763] LustreError: 329007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782862566 with bad export cookie 10713866319897529761 [ 5304.045110] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5304.047410] LustreError: 329007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5304.196297] Lustre: server umount lustre-MDT0001 complete [ 5311.331557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5311.499254] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5311.522849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5311.637410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5313.594066] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5317.958798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5318.009381] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5318.031280] Lustre: 331612:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5318.206333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5320.239070] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5323.807609] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5326.583096] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5332.453335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5332.551520] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5332.555515] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5332.559161] LustreError: 333496:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5332.561550] LustreError: 333496:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5332.564594] LustreError: 333496:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5332.599769] Lustre: server umount lustre-OST0000 complete [ 5332.601968] LustreError: 333496:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5338.725029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5338.834076] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5338.945732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5341.559055] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5345.035247] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5346.356443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5348.741987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5348.799688] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5348.857803] Lustre: lustre-OST0001: new disk, initializing [ 5348.861296] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5348.866442] Lustre: 335038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5348.921667] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5351.655349] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5352.675501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5352.682168] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5352.722973] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5354.751917] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5358.054305] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5358.056884] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.059353] LustreError: Skipped 1 previous similar message [ 5358.060290] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5358.067190] Lustre: Skipped 1 previous similar message [ 5362.316935] Lustre: server umount lustre-OST0001 complete [ 5363.175307] LustreError: 334735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5363.185436] LustreError: 334735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5366.752930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5366.758047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.767298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5366.771314] Lustre: Skipped 2 previous similar messages [ 5368.295082] LustreError: 334735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5368.305257] LustreError: 334735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5370.188574] Lustre: server umount lustre-OST0000 complete [ 5373.410195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5373.418959] Lustre: Skipped 2 previous similar messages [ 5383.649160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5383.653391] Lustre: Skipped 7 previous similar messages [ 5386.207126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5386.300884] Lustre: server umount lustre-MDT0000 complete [ 5388.074193] LustreError: 331604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782862650 with bad export cookie 10713866319897530300 [ 5388.079278] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5388.081597] LustreError: 331604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5388.206798] Lustre: server umount lustre-MDT0001 complete [ 5393.069314] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5394.523128] Key type lgssc unregistered [ 5394.679636] LNet: 336994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5394.684781] LNetError: 336994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5394.693557] LNet: Removed LNI 192.168.204.113@tcp [ 5395.143196] Key type .llcrypt unregistered [ 5395.145531] Key type ._llcrypt unregistered [ 5429.854401] Key type ._llcrypt registered [ 5429.855710] Key type .llcrypt registered [ 5429.896936] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 5437.875952] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5438.198461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5438.267617] alg: No test for adler32 (adler32-zlib) [ 5439.125436] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 5439.213073] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5440.799159] Key type lgssc registered [ 5441.337058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5456.980569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5456.988183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5458.081568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5458.112324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5458.161326] Lustre: lustre-MDT0000: new disk, initializing [ 5458.185683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5458.194155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5459.922712] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5465.498141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5465.569367] Lustre: 341896:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5465.613214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5465.617419] Lustre: Skipped 1 previous similar message [ 5465.666042] Lustre: lustre-MDT0001: new disk, initializing [ 5465.706718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5465.721954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5465.726231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5467.247260] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5471.308157] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5473.889993] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5476.700110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5476.823545] Lustre: lustre-OST0000: new disk, initializing [ 5476.825384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5476.828144] Lustre: 343865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5476.855488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5478.878115] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5482.890493] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5484.391817] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5487.113898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5487.120390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5487.161946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5487.572941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5489.026064] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5489.093124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5492.197561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5492.199514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5492.206896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5492.214765] Lustre: Skipped 1 previous similar message [ 5495.974259] Lustre: server umount lustre-OST0000 complete [ 5501.408439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5501.413499] LustreError: Skipped 1 previous similar message [ 5501.416466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5501.423939] Lustre: Skipped 1 previous similar message [ 5501.427163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5502.433139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5502.434295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.449611] Lustre: server umount lustre-MDT0000 complete [ 5505.005474] LustreError: 341889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782862767 with bad export cookie 17155147059016305308 [ 5505.010531] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5505.014238] LustreError: 341889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5505.125077] Lustre: server umount lustre-MDT0001 complete [ 5508.402164] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:39:29 (1782862769) [ 5511.053435] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:39:32 (1782862772) [ 5514.340948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5514.508549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.624828] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5518.205921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5519.462671] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5521.734947] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5524.327767] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5527.075295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5527.236246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5527.239554] Lustre: Skipped 1 previous similar message [ 5529.832583] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5532.722413] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5546.822976] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5547.297203] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5547.751591] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5548.213266] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5548.780084] Lustre: Failing over lustre-MDT0000 [ 5549.024105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5549.026733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5549.027981] Lustre: server umount lustre-MDT0000 complete [ 5549.030285] LustreError: 346640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5549.035479] LustreError: 346640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5550.048286] LustreError: 346639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5550.052474] LustreError: 346639:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5553.633285] LustreError: 346635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5558.748186] LustreError: 346636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5558.753429] LustreError: 346636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5561.570927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5561.605988] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5561.691274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5561.709195] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5562.836418] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5563.868196] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5566.949334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5566.953382] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5566.968025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5568.475940] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5568.944384] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5570.706232] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5571.298841] Lustre: Failing over lustre-MDT0000 [ 5571.470967] Lustre: server umount lustre-MDT0000 complete [ 5572.063709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5572.065877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5572.069050] Lustre: Skipped 2 previous similar messages [ 5572.071108] LustreError: 346658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5572.075516] LustreError: 346658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5582.304351] LustreError: 346658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5582.308367] LustreError: 346658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5584.009305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5584.046397] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5584.126699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.148342] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5584.347798] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5585.310499] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5589.476023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5589.477764] Lustre: Skipped 2 previous similar messages [ 5589.485929] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5589.500918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5591.000623] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5591.492439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5594.592551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5594.594324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5594.594559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5594.594569] Lustre: Skipped 1 previous similar message [ 5594.595763] Lustre: Skipped 4 previous similar messages [ 5599.712395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5599.715109] Lustre: Skipped 4 previous similar messages [ 5599.770535] Lustre: server umount lustre-MDT0000 complete [ 5600.872947] LustreError: 349431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782862863 with bad export cookie 17155147059016307961 [ 5600.875180] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5600.876371] LustreError: 349431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5600.983729] Lustre: server umount lustre-MDT0001 complete [ 5612.498331] Lustre: server umount lustre-OST0000 complete [ 5617.003040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5617.147485] LustreError: 352988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5617.169748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5618.377210] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5620.906944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5622.203205] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5625.017736] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5627.273917] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5629.669582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5631.445912] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5633.568114] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5642.091887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5646.303757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5646.305825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.309321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5651.592975] Lustre: server umount lustre-OST0000 complete [ 5662.687972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5662.689867] Lustre: Skipped 5 previous similar messages [ 5667.295102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5667.374617] Lustre: server umount lustre-MDT0000 complete [ 5667.808789] LustreError: 353721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.818610] LustreError: 353721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5668.966936] LustreError: 352968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782862931 with bad export cookie 17155147059016308696 [ 5668.974109] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5668.974443] LustreError: 352968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5669.116126] Lustre: server umount lustre-MDT0001 complete [ 5673.307743] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5674.600686] Key type lgssc unregistered [ 5674.731421] LNet: 356539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5674.735455] LNetError: 356539:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5674.747428] LNet: Removed LNI 192.168.204.113@tcp [ 5675.072572] Key type .llcrypt unregistered [ 5675.073926] Key type ._llcrypt unregistered [ 5686.511632] Key type ._llcrypt registered [ 5686.512709] Key type .llcrypt registered [ 5686.553526] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5686.946439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5686.952790] alg: No test for adler32 (adler32-zlib) [ 5687.797418] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 5687.878832] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5689.455194] Key type lgssc registered [ 5690.032071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5693.319411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5693.330983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5694.486983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5696.109362] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5700.120851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5700.298485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5701.931883] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5705.335319] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5707.369280] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5710.056582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5710.162198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5711.955540] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5715.086777] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5719.406040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5724.640653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5724.642199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5724.644422] Lustre: Skipped 2 previous similar messages [ 5724.647417] Lustre: Skipped 2 previous similar messages [ 5729.760198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5729.761946] Lustre: Skipped 1 previous similar message [ 5730.385616] Lustre: server umount lustre-MDT0000 complete [ 5731.484088] LustreError: 358014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782862993 with bad export cookie 10550088835713215698 [ 5731.486843] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5731.487142] LustreError: 358014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5731.587735] Lustre: server umount lustre-MDT0001 complete [ 5742.999518] Lustre: server umount lustre-OST0000 complete [ 5746.979099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.117430] LustreError: 361742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5747.122969] LustreError: 361742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5747.137383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5748.321510] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5750.893199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5752.299326] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5754.601556] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5756.290928] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5758.440383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5758.540180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5758.541992] Lustre: Skipped 1 previous similar message [ 5760.176808] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5762.220509] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5771.882749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5781.984462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5781.989648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5781.998966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5785.711940] Lustre: server umount lustre-MDT0000 complete [ 5787.263921] LustreError: 361723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782863049 with bad export cookie 10550088835713216797 [ 5787.268629] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5787.271175] LustreError: 361723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5787.433192] Lustre: server umount lustre-MDT0001 complete [ 5799.411175] Lustre: server umount lustre-OST0000 complete [ 5806.623474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5806.853168] LustreError: 365669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5806.889507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5808.503054] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5811.487446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5812.857211] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5815.018817] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5816.625650] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5818.688748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5821.025921] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5823.570842] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5832.104530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5836.255862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5836.258234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5836.261333] Lustre: Skipped 2 previous similar messages [ 5836.262848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5836.264490] Lustre: Skipped 3 previous similar messages [ 5837.284445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5841.504284] Lustre: server umount lustre-OST0000 complete [ 5847.522165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5847.522319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.523339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5847.523351] Lustre: Skipped 2 previous similar messages [ 5847.541921] Lustre: Skipped 2 previous similar messages [ 5857.247141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5857.328823] Lustre: server umount lustre-MDT0000 complete [ 5857.760793] LustreError: 368157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5857.771885] LustreError: 368157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5859.101868] LustreError: 365649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782863121 with bad export cookie 10550088835713217805 [ 5859.107767] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5859.109416] LustreError: 365649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5859.264099] Lustre: server umount lustre-MDT0001 complete [ 5863.995520] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5865.472737] Key type lgssc unregistered [ 5865.642543] LNet: 369224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5865.647512] LNetError: 369224:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5865.661497] LNet: Removed LNI 192.168.204.113@tcp [ 5866.098221] Key type .llcrypt unregistered [ 5866.100404] Key type ._llcrypt unregistered [ 5878.405446] Key type ._llcrypt registered [ 5878.406869] Key type .llcrypt registered [ 5878.447084] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5878.842570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5878.874747] alg: No test for adler32 (adler32-zlib) [ 5879.730875] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 5879.811845] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5881.383158] Key type lgssc registered [ 5881.961004] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5885.739201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5885.749925] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5885.755721] LustreError: 370641:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5885.767509] LustreError: 370641:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5885.772193] LustreError: 370641:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5885.776944] LustreError: 370641:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5885.782170] LustreError: 370641:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5889.323127] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5889.325740] LustreError: 371038:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5889.334967] LustreError: 371038:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5889.337335] LustreError: 371038:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5889.339872] LustreError: 371038:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5889.342468] LustreError: 371038:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5897.435451] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 5898.833377] Key type lgssc unregistered [ 5898.989634] LNet: 372039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5898.994486] LNetError: 372039:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5899.006571] LNet: Removed LNI 192.168.204.113@tcp [ 5899.438509] Key type .llcrypt unregistered [ 5899.440820] Key type ._llcrypt unregistered [ 5907.463910] Key type ._llcrypt registered [ 5907.465599] Key type .llcrypt registered [ 5907.522536] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 5915.789937] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5916.112172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5916.137592] alg: No test for adler32 (adler32-zlib) [ 5916.991272] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 5917.083751] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 5918.663145] Key type lgssc registered [ 5919.195758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5935.734229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5935.746464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5936.860698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5936.875126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5936.917518] Lustre: lustre-MDT0000: new disk, initializing [ 5936.942175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5936.949023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5938.722869] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5944.306216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.369519] Lustre: 375170:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5944.411266] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5944.415389] Lustre: Skipped 1 previous similar message [ 5944.461722] Lustre: lustre-MDT0001: new disk, initializing [ 5944.504363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5944.524698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5944.531146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5946.296692] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5950.790536] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5953.285912] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5956.330882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5956.428587] Lustre: lustre-OST0000: new disk, initializing [ 5956.430291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5956.432689] Lustre: 377137:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5956.463261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5957.113307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5957.117410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5957.148398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5958.466544] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5963.264219] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5965.078305] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5965.190154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5967.126314] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5967.247347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5972.452301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5972.453448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5972.457307] LustreError: Skipped 1 previous similar message [ 5972.466992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5972.471898] Lustre: Skipped 1 previous similar message [ 5974.250356] Lustre: server umount lustre-OST0000 complete [ 5977.570212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.571563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5977.578362] Lustre: Skipped 2 previous similar messages [ 5982.688978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5982.692904] Lustre: Skipped 2 previous similar messages [ 5987.809091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5987.813129] Lustre: Skipped 1 previous similar message [ 5990.367140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5990.458380] Lustre: server umount lustre-MDT0000 complete [ 5992.138603] LustreError: 375160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782863254 with bad export cookie 17052802323230933171 [ 5992.143106] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5992.147607] LustreError: 375160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5992.306675] Lustre: server umount lustre-MDT0001 complete [ 5996.000349] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5996.820748] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:47:38 (1782863258) [ 5999.909033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6000.118319] LustreError: 379595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6000.127655] LustreError: 379595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6000.150781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.825746] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6005.216684] LustreError: 379596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.794921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.942872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6007.644346] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6010.783966] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6012.873126] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6015.979924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6016.119147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6018.612462] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6021.845367] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6025.097031] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6025.206539] Lustre: lustre-OST0001: new disk, initializing [ 6025.210181] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6025.215243] Lustre: 382432:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6025.505357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6025.512139] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6025.565272] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6027.882659] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6032.732606] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6035.937480] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6035.937763] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6035.942532] LustreError: Skipped 1 previous similar message [ 6035.951448] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6035.957527] Lustre: Skipped 1 previous similar message [ 6041.036703] Lustre: server umount lustre-OST0001 complete [ 6042.084636] LustreError: 381557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6042.092167] LustreError: 381557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6047.207849] LustreError: 381418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6047.218194] LustreError: 381418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6052.328591] LustreError: 381419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.338973] LustreError: 381419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6057.443239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6057.444406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6057.448772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6057.448785] Lustre: Skipped 3 previous similar messages [ 6057.451176] Lustre: Skipped 2 previous similar messages [ 6059.812148] Lustre: server umount lustre-OST0000 complete [ 6062.563175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6062.563478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6062.576207] Lustre: Skipped 1 previous similar message [ 6067.887447] Lustre: server umount lustre-MDT0000 complete [ 6069.636859] LustreError: 380331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782863332 with bad export cookie 17052802323230934053 [ 6069.644199] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6069.644439] LustreError: 380331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6075.857390] Lustre: server umount lustre-MDT0001 complete [ 6080.884750] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6082.365412] Key type lgssc unregistered [ 6082.531620] LNet: 384416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6082.536900] LNetError: 384416:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6082.548596] LNet: Removed LNI 192.168.204.113@tcp [ 6082.984969] Key type .llcrypt unregistered [ 6082.987251] Key type ._llcrypt unregistered [ 6108.556551] Key type ._llcrypt registered [ 6108.558305] Key type .llcrypt registered [ 6108.612750] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6109.034858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6109.060075] alg: No test for adler32 (adler32-zlib) [ 6109.972408] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 6110.099418] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6111.719154] Key type lgssc registered [ 6112.367881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6116.428004] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6116.440243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.573905] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6117.587687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6117.672258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6119.551310] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6123.789136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6123.840625] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6123.855703] Lustre: 387190:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6123.998967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6125.902562] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6129.300861] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6131.649405] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6134.240433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6134.245424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6134.255147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6135.777119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6135.778228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6138.784880] Lustre: server umount lustre-MDT0000 complete [ 6140.257321] LustreError: 387183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782863402 with bad export cookie 602410299566094823 [ 6140.260837] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6140.261412] LustreError: 387183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6140.381363] Lustre: server umount lustre-MDT0001 complete [ 6144.275323] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:50:05 (1782863405) [ 6146.665043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6146.822981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6148.231933] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6151.765955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6151.910208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6153.796517] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6156.877246] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6158.696252] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6160.678352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6160.738595] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6160.791479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6162.464704] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6164.613536] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6168.210671] Lustre: Permanently deactivating lustre-OST0000 [ 6168.215179] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6169.261883] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6170.057649] LustreError: 392542:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6170.061349] LustreError: 392542:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6170.064732] LustreError: 392542:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6170.613736] LustreError: 392590:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6170.616379] LustreError: 392590:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6177.289812] Lustre: server umount lustre-OST0000 complete [ 6178.099767] LustreError: 392839:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6178.102970] LustreError: 392839:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6182.879751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6182.882024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6182.885832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6182.887568] Lustre: Skipped 1 previous similar message [ 6188.511945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6188.513977] Lustre: Skipped 3 previous similar messages [ 6189.559293] Lustre: server umount lustre-MDT0000 complete [ 6190.642079] LustreError: 389684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782863453 with bad export cookie 602410299566095390 [ 6190.644976] LustreError: 389684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6190.645406] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6200.842902] Lustre: server umount lustre-MDT0001 complete [ 6203.854575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6205.453960] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6206.744774] LustreError: 393996:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6209.413561] Lustre: 394335:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.113@tcp [ 6210.197929] Lustre: server umount MGS complete [ 6214.682619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.930585] LustreError: 395058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6214.940502] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6214.943880] LustreError: 395058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6214.946971] Lustre: Skipped 1 previous similar message [ 6214.992902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6216.995714] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6220.258587] LustreError: 395059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6221.004987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6221.144163] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6223.051881] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6226.517431] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6229.042445] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6229.968879] Lustre: Permanently reactivating lustre-OST0000 [ 6229.979392] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6229.983871] Lustre: Skipped 2 previous similar messages [ 6253.535220] LustreError: 396759:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6257.480300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6257.597143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6257.599834] Lustre: Skipped 1 previous similar message [ 6260.059936] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6263.459603] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6272.996208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6272.996627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6273.000886] LustreError: Skipped 1 previous similar message [ 6273.007087] Lustre: Skipped 1 previous similar message [ 6273.007969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6273.015376] Lustre: Skipped 1 previous similar message [ 6276.633215] Lustre: server umount lustre-OST0000 complete [ 6282.720468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6282.725611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6282.733359] Lustre: Skipped 1 previous similar message [ 6282.736881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6284.398138] Lustre: server umount lustre-MDT0000 complete [ 6286.178858] LustreError: 395038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782863548 with bad export cookie 602410299566096426 [ 6286.185639] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6286.186504] LustreError: 395038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6286.333183] Lustre: server umount lustre-MDT0001 complete [ 6291.318270] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6292.835642] Key type lgssc unregistered [ 6293.018443] LNet: 399562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6293.021925] LNetError: 399562:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6293.031479] LNet: Removed LNI 192.168.204.113@tcp [ 6293.497164] Key type .llcrypt unregistered [ 6293.499511] Key type ._llcrypt unregistered [ 6301.015106] Key type ._llcrypt registered [ 6301.016240] Key type .llcrypt registered [ 6301.047830] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 6309.084594] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6309.415799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6309.460614] alg: No test for adler32 (adler32-zlib) [ 6310.328432] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 6310.429398] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6312.007197] Key type lgssc registered [ 6312.556767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6329.076686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6329.087608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.186750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6330.199263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6330.240802] Lustre: lustre-MDT0000: new disk, initializing [ 6330.265916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6330.275045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6331.605561] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6336.024690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6336.062385] Lustre: 402693:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6336.087730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6336.090361] Lustre: Skipped 1 previous similar message [ 6336.129160] Lustre: lustre-MDT0001: new disk, initializing [ 6336.163821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6336.177342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6336.180902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6337.877552] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6342.691173] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6345.230899] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6348.398267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6348.558298] Lustre: lustre-OST0000: new disk, initializing [ 6348.561599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6348.566600] Lustre: 404662:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6348.609287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6351.220565] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6354.440165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6354.446787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6354.504639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6356.173836] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6358.070171] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6358.179098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6360.043165] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6360.164456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6364.645795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6364.646529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.650863] LustreError: Skipped 1 previous similar message [ 6364.659922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6364.665614] Lustre: Skipped 1 previous similar message [ 6369.764200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6369.768267] Lustre: Skipped 1 previous similar message [ 6374.885815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6374.889792] Lustre: Skipped 1 previous similar message [ 6375.391140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6375.472498] Lustre: server umount lustre-OST0000 complete [ 6377.440470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6377.445604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6377.453454] Lustre: Skipped 1 previous similar message [ 6377.457083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6380.002888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6383.410213] Lustre: server umount lustre-MDT0000 complete [ 6385.160055] LustreError: 402686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782863647 with bad export cookie 15708305812510220301 [ 6385.167124] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6385.167747] LustreError: 402686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6385.296554] Lustre: server umount lustre-MDT0001 complete [ 6389.237868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6390.087980] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:54:11 (1782863651) [ 6393.254566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6393.475352] LustreError: 407120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6393.486771] LustreError: 407120:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6393.514739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6395.411666] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6398.944935] LustreError: 407121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.485884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6399.636923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6401.287713] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6404.773141] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6407.263772] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6410.241770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6410.416861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6412.923087] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6416.188790] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6417.120611] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6424.118204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6429.155333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6429.158620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6429.164515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6429.168570] LustreError: Skipped 1 previous similar message [ 6429.172870] Lustre: Skipped 2 previous similar messages [ 6432.099708] Lustre: server umount lustre-OST0000 complete [ 6434.274792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6434.282971] Lustre: Skipped 2 previous similar messages [ 6439.393109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6439.397021] Lustre: Skipped 3 previous similar messages [ 6440.094501] Lustre: server umount lustre-MDT0000 complete [ 6441.501848] LustreError: 407100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782863703 with bad export cookie 15708305812510221155 [ 6441.506166] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6441.509174] LustreError: 407100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6441.658657] Lustre: server umount lustre-MDT0001 complete [ 6446.091655] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6447.465292] Key type lgssc unregistered [ 6447.635627] LNet: 410846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6447.640383] LNetError: 410846:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6447.654448] LNet: Removed LNI 192.168.204.113@tcp [ 6448.115034] Key type .llcrypt unregistered [ 6448.117421] Key type ._llcrypt unregistered [ 6457.983861] Key type ._llcrypt registered [ 6457.985273] Key type .llcrypt registered [ 6458.030909] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6458.183494] Key type .llcrypt unregistered [ 6458.185708] Key type ._llcrypt unregistered [ 6465.970277] Key type ._llcrypt registered [ 6465.971977] Key type .llcrypt registered [ 6466.030684] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6466.333352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6466.367589] alg: No test for adler32 (adler32-zlib) [ 6467.242942] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 6467.350062] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6468.943182] Key type lgssc registered [ 6469.394685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6472.218787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6472.225225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6473.369119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6474.652955] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6477.796747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6477.915432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6479.683252] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6481.739633] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6482.913281] LustreError: 413034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.917772] LustreError: 413034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6483.679750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.793610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6484.838084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6484.931390] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6489.056221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6489.059569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.063445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6490.082287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6490.083161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6490.083175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6490.088852] Lustre: Skipped 1 previous similar message [ 6493.652522] Lustre: server umount lustre-OST0000 complete [ 6495.200886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6495.207724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6495.211443] Lustre: Skipped 1 previous similar message [ 6500.321066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6500.325209] Lustre: Skipped 1 previous similar message [ 6501.221854] Lustre: server umount lustre-MDT0000 complete [ 6503.034825] LustreError: 413014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782863765 with bad export cookie 17318201973943026558 [ 6503.041268] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6503.042936] LustreError: 413014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6503.182575] Lustre: server umount lustre-MDT0001 complete [ 6506.745967] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6507.681075] Key type lgssc unregistered [ 6507.819680] LNet: 416177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6507.824589] LNetError: 416177:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6507.836589] LNet: Removed LNI 192.168.204.113@tcp [ 6508.264555] Key type .llcrypt unregistered [ 6508.266213] Key type ._llcrypt unregistered [ 6519.040098] Key type ._llcrypt registered [ 6519.041439] Key type .llcrypt registered [ 6519.089644] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6519.425035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6519.447196] alg: No test for adler32 (adler32-zlib) [ 6520.319067] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 6520.417922] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6522.007195] Key type lgssc registered [ 6522.592672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6526.231564] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6526.243205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6541.727485] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6546.847369] LustreError: 417544:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [ 6573.471235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6576.068920] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6579.287038] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6582.396891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6582.609883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6582.653811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6583.880093] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6586.311723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6586.425143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6587.545271] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6589.590653] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6591.114293] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6593.505970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6593.506728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6593.509697] Lustre: Skipped 1 previous similar message [ 6598.628399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6598.632187] Lustre: Skipped 2 previous similar messages [ 6599.335523] Lustre: server umount lustre-OST0000 complete [ 6601.696399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6601.701348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6601.709723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6603.746894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6603.747240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6607.207053] Lustre: server umount lustre-MDT0000 complete [ 6608.681935] LustreError: 417550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782863871 with bad export cookie 16125475341291249060 [ 6608.684746] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6608.685342] LustreError: 417550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6608.799767] Lustre: server umount lustre-MDT0001 complete [ 6612.030646] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6613.389343] Key type lgssc unregistered [ 6613.553573] LNet: 421112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6613.558369] LNetError: 421112:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6613.569527] LNet: Removed LNI 192.168.204.113@tcp [ 6613.943062] Key type .llcrypt unregistered [ 6613.945322] Key type ._llcrypt unregistered [ 6624.611746] Key type ._llcrypt registered [ 6624.612873] Key type .llcrypt registered [ 6624.648438] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6624.993330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6624.999960] alg: No test for adler32 (adler32-zlib) [ 6625.841758] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 6625.935785] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6627.535165] Key type lgssc registered [ 6628.062430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6631.090390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6631.101125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6632.252151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6633.752665] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6637.061745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6637.204926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6638.895239] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6642.014258] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6644.230457] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6647.172756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6647.352601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6649.740962] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6652.787352] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6657.580509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6662.627565] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6662.630528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6662.638485] Lustre: Skipped 2 previous similar messages [ 6662.645016] Lustre: Skipped 2 previous similar messages [ 6663.140628] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 6665.137304] Lustre: server umount lustre-MDT0000 complete [ 6666.172099] LustreError: 422484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.13@tcp arrived at 1782863928 with bad export cookie 4022563558688325729 [ 6666.181230] LustreError: 422484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6667.085559] LustreError: 423240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782863929 with bad export cookie 4022563558688325078 [ 6667.092135] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6667.092895] LustreError: 423240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6667.744792] LustreError: 422504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6667.746037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.754721] LustreError: 422504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6667.763512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6672.863839] LustreError: 422523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6672.864114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6672.867895] LustreError: 422523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6673.227900] Lustre: server umount lustre-OST0000 complete [ 6675.331763] Lustre: server umount lustre-MDT0001 complete [ 6678.797282] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6679.677734] Key type lgssc unregistered [ 6679.791336] LNet: 426192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6679.793352] LNetError: 426192:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6679.805269] LNet: Removed LNI 192.168.204.113@tcp [ 6680.098820] Key type .llcrypt unregistered [ 6680.099888] Key type ._llcrypt unregistered [ 6688.817251] Key type ._llcrypt registered [ 6688.818210] Key type .llcrypt registered [ 6688.848436] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6689.242132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6689.250206] alg: No test for adler32 (adler32-zlib) [ 6690.095095] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 6690.171562] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6691.743087] Key type lgssc registered [ 6692.056669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6694.379880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6694.385503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6695.518758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6696.698226] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6699.243136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6699.368025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6700.544468] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6702.708687] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6704.355842] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6706.384491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6706.491502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6708.275037] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6710.681860] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6720.871965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6725.088341] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6725.092027] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.098118] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6726.112881] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6726.114531] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6728.526920] Lustre: server umount lustre-MDT0001 complete [ 6731.233057] LustreError: 427584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6731.233926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6731.234225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6731.235401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6731.235406] Lustre: Skipped 2 previous similar messages [ 6731.240285] LustreError: 427584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6736.352897] LustreError: 427602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6736.353818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6736.410960] Lustre: server umount lustre-OST0000 complete [ 6737.648540] Lustre: server umount lustre-MDT0000 complete [ 6741.930235] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6742.812461] Key type lgssc unregistered [ 6742.925399] LNet: 431288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6742.927683] LNetError: 431288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6742.934301] LNet: Removed LNI 192.168.204.113@tcp [ 6743.214532] Key type .llcrypt unregistered [ 6743.215635] Key type ._llcrypt unregistered [ 6750.876213] Key type ._llcrypt registered [ 6750.877326] Key type .llcrypt registered [ 6750.908553] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 6756.965335] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6757.265937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6757.273496] alg: No test for adler32 (adler32-zlib) [ 6758.107584] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 6758.196433] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6759.767089] Key type lgssc registered [ 6760.146273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6771.952064] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6776.786645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6776.791215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6777.864323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6777.872896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6777.906743] Lustre: lustre-MDT0000: new disk, initializing [ 6777.925644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6777.930661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6779.082478] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6784.329701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6784.421316] Lustre: lustre-OST0000: new disk, initializing [ 6784.423645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6784.426259] Lustre: Skipped 1 previous similar message [ 6784.428736] Lustre: 436292:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6784.450539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6786.149815] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6787.514698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6787.517361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6787.525244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6789.638588] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6794.695664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6794.731733] Lustre: 435275:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6794.760671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6794.797994] Lustre: lustre-MDT0001: new disk, initializing [ 6794.835432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6794.847109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6794.849941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6796.013012] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6799.281958] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6799.341595] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6800.503785] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6800.568201] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6803.939214] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.13@tcp (stopping) [ 6804.046306] Lustre: server umount lustre-MDT0001 complete [ 6805.473464] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6805.473784] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6810.593034] LustreError: 435281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.597082] LustreError: 435281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6815.714970] LustreError: 437004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6815.721524] LustreError: 437004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6818.271812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6818.273853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.276904] Lustre: Skipped 1 previous similar message [ 6818.278378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6820.832563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6820.833439] LustreError: 435285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6822.346952] Lustre: server umount lustre-OST0000 complete [ 6829.807770] Lustre: server umount lustre-MDT0000 complete [ 6833.903509] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6834.802740] Key type lgssc unregistered [ 6834.918353] LNet: 440048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6834.920682] LNetError: 440048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6834.928268] LNet: Removed LNI 192.168.204.113@tcp [ 6835.201145] Key type .llcrypt unregistered [ 6835.202185] Key type ._llcrypt unregistered [ 6847.397973] Key type ._llcrypt registered [ 6847.398962] Key type .llcrypt registered [ 6847.437182] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6847.531287] Key type .llcrypt unregistered [ 6847.532276] Key type ._llcrypt unregistered [ 6853.851541] Key type ._llcrypt registered [ 6853.852528] Key type .llcrypt registered [ 6853.889527] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6854.268596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6854.324119] alg: No test for adler32 (adler32-zlib) [ 6855.173239] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 6855.266813] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6856.847196] Key type lgssc registered [ 6857.192819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6865.339909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6865.346202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.426395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6866.436427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6866.470220] Lustre: lustre-MDT0000: new disk, initializing [ 6866.490723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6866.496355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6867.777311] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6871.733484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6871.773580] Lustre: 443392:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6871.820810] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6871.824066] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6871.825646] Lustre: Skipped 1 previous similar message [ 6871.863879] Lustre: lustre-MDT0001: new disk, initializing [ 6871.897511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6871.913545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6871.917612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6873.155194] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6876.692489] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6878.417889] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6880.541546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6880.646248] Lustre: lustre-OST0000: new disk, initializing [ 6880.647863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6880.650240] Lustre: 445359:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6880.680579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6882.491905] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6885.381029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6885.384862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6885.432530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6886.077552] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6890.467443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6890.469789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6890.472182] Lustre: Skipped 1 previous similar message [ 6890.474881] Lustre: Skipped 2 previous similar messages [ 6894.633447] Lustre: server umount lustre-MDT0000 complete [ 6895.813271] LustreError: 443387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782864158 with bad export cookie 11100570273383807436 [ 6895.816734] LustreError: 443387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6895.818405] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6895.933288] Lustre: server umount lustre-MDT0001 complete [ 6907.359990] Lustre: server umount lustre-OST0000 complete [ 6910.599848] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:02:52 (1782864172) [ 6913.036070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.204274] LustreError: 447386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6913.209218] LustreError: 447386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6913.227309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6914.426436] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6917.091412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.478685] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6920.766169] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6922.502046] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6937.762047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6937.828116] Lustre: Found index 0 for lustre-OST0000, updating log [ 6937.888335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6937.890137] Lustre: Skipped 1 previous similar message [ 6938.921270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6939.716461] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6942.312058] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6944.224195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6944.229249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6944.237022] Lustre: Skipped 1 previous similar message [ 6944.240315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6947.298069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6947.300268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6947.304844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6949.837189] Lustre: server umount lustre-OST0000 complete [ 6962.354787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6962.355257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6962.362644] Lustre: Skipped 1 previous similar message [ 6962.364659] Lustre: Skipped 2 previous similar messages [ 6968.482334] Lustre: server umount lustre-MDT0000 complete [ 6969.863068] LustreError: 449229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782864232 with bad export cookie 11100570273383808829 [ 6969.865817] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6969.866708] LustreError: 449229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6969.989931] Lustre: server umount lustre-MDT0001 complete [ 6973.129431] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:03:54 (1782864234) [ 7004.767301] LNet: 1 local NIs in recovery (showing 1): 192.168.204.113@tcp [ 7559.129298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7559.299897] LustreError: 453788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7559.320278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7560.477324] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7563.000245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7563.117442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7564.277134] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7566.354436] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7567.922839] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7583.010979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7583.116090] Lustre: Found index 0 for lustre-OST0000, updating log [ 7583.200697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7584.944148] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7586.285192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 7587.416543] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7589.220317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.228584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7589.232275] Lustre: Skipped 3 previous similar messages [ 7594.344485] Lustre: server umount lustre-OST0000 complete [ 7596.122352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7596.123797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7596.130216] Lustre: Skipped 1 previous similar message [ 7598.177105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7598.181109] Lustre: Skipped 4 previous similar messages [ 7602.273263] LustreError: 453789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7602.281851] LustreError: 453789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7602.297174] Lustre: server umount lustre-MDT0000 complete [ 7603.297077] LustreError: 453807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7603.308552] LustreError: 453807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7604.101257] LustreError: 453768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782864866 with bad export cookie 11100570273383809900 [ 7604.106511] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7604.108746] LustreError: 453768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7604.255539] Lustre: server umount lustre-MDT0001 complete [ 7618.144264] LNet: 1 local NIs in recovery (showing 1): 192.168.204.113@tcp [ 7624.118344] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 7628.329322] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7644.353987] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:15:05 (1782864905) [ 7651.315100] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7652.870381] Key type lgssc unregistered [ 7653.054733] LNet: 460444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7653.058277] LNetError: 460444:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7714.533586] LNet: Removed LNI 192.168.204.113@tcp [ 7715.048257] Key type .llcrypt unregistered [ 7715.050820] Key type ._llcrypt unregistered [ 7723.742884] Key type ._llcrypt registered [ 7723.744498] Key type .llcrypt registered [ 7723.812651] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7724.368053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7724.380316] alg: No test for adler32 (adler32-zlib) [ 7725.267501] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 7725.389256] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 7726.999221] Key type lgssc registered [ 7727.586907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7729.862106] LNet: Added LNI 192.168.204.113@tcp100 [8/256/0/180] [ 7730.299593] LNet: Added LNI 192.168.204.113@tcp101 [8/256/0/180] [ 7730.737984] LNet: Added LNI 192.168.204.113@tcp102 [8/256/0/180] [ 7731.183202] LNet: Added LNI 192.168.204.113@tcp103 [8/256/0/180] [ 7731.642941] LNet: Added LNI 192.168.204.113@tcp104 [8/256/0/180] [ 7732.091376] LNet: Added LNI 192.168.204.113@tcp105 [8/256/0/180] [ 7732.557149] LNet: Added LNI 192.168.204.113@tcp106 [8/256/0/180] [ 7732.993161] LNet: Added LNI 192.168.204.113@tcp107 [8/256/0/180] [ 7733.430393] LNet: Added LNI 192.168.204.113@tcp108 [8/256/0/180] [ 7733.875839] LNet: Added LNI 192.168.204.113@tcp109 [8/256/0/180] [ 7734.323440] LNet: Added LNI 192.168.204.113@tcp110 [8/256/0/180] [ 7734.784766] LNet: Added LNI 192.168.204.113@tcp111 [8/256/0/180] [ 7735.231565] LNet: Added LNI 192.168.204.113@tcp112 [8/256/0/180] [ 7735.665833] LNet: Added LNI 192.168.204.113@tcp113 [8/256/0/180] [ 7736.117795] LNet: Added LNI 192.168.204.113@tcp114 [8/256/0/180] [ 7736.562826] LNet: Added LNI 192.168.204.113@tcp115 [8/256/0/180] [ 7737.004084] LNet: Added LNI 192.168.204.113@tcp116 [8/256/0/180] [ 7737.450303] LNet: Added LNI 192.168.204.113@tcp117 [8/256/0/180] [ 7737.902885] LNet: Added LNI 192.168.204.113@tcp118 [8/256/0/180] [ 7738.322864] LNet: Added LNI 192.168.204.113@tcp119 [8/256/0/180] [ 7738.758463] LNet: Added LNI 192.168.204.113@tcp120 [8/256/0/180] [ 7739.205544] LNet: Added LNI 192.168.204.113@tcp121 [8/256/0/180] [ 7739.641259] LNet: Added LNI 192.168.204.113@tcp122 [8/256/0/180] [ 7740.058232] LNet: Added LNI 192.168.204.113@tcp123 [8/256/0/180] [ 7740.464783] LNet: Added LNI 192.168.204.113@tcp124 [8/256/0/180] [ 7740.867357] LNet: Added LNI 192.168.204.113@tcp125 [8/256/0/180] [ 7741.309835] LNet: Added LNI 192.168.204.113@tcp126 [8/256/0/180] [ 7741.702533] LNet: Added LNI 192.168.204.113@tcp127 [8/256/0/180] [ 7742.131773] LNet: Added LNI 192.168.204.113@tcp128 [8/256/0/180] [ 7742.561084] LNet: Added LNI 192.168.204.113@tcp129 [8/256/0/180] [ 7742.944944] LNet: Added LNI 192.168.204.113@tcp130 [8/256/0/180] [ 7743.298908] LNet: Added LNI 192.168.204.113@tcp131 [8/256/0/180] [ 7743.719351] LNet: Added LNI 192.168.204.113@tcp132 [8/256/0/180] [ 7744.129458] LNet: Added LNI 192.168.204.113@tcp133 [8/256/0/180] [ 7744.556332] LNet: Added LNI 192.168.204.113@tcp134 [8/256/0/180] [ 7744.987624] LNet: Added LNI 192.168.204.113@tcp135 [8/256/0/180] [ 7748.232235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7748.241224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7749.289534] Lustre: 463441:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.113@tcp: can't add NID 192.168.204.113@tcp130: rc = -75 [ 7749.319881] LNetError: 463464:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [ 7749.340389] LNetError: Protocol error connecting to 192.168.204.113@tcp111 at host 192.168.204.113:988: is it running a compatible version of Lustre? [ 7749.343499] LNetError: Skipped 1 previous similar message [ 7749.393414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7749.405772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7749.446220] Lustre: lustre-MDT0000: new disk, initializing [ 7749.477506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7749.488140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7749.983086] LNet: 4 local NIs in recovery (showing 4): 192.168.204.113@tcp126, 192.168.204.113@tcp127, 192.168.204.113@tcp129, 192.168.204.113@tcp128 [ 7750.376162] LNetError: 463469:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [ 7750.383903] LNetError: 463469:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 7750.389870] LNetError: Protocol error connecting to 192.168.204.113@tcp103 at host 192.168.204.113:988: is it running a compatible version of Lustre? [ 7750.398051] LNetError: Skipped 6 previous similar messages [ 7751.187816] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7753.060553] LNetError: 460906:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [ 7753.065234] LNetError: 460906:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [ 7753.069736] LNetError: Protocol error connecting to 192.168.204.113@tcp126 at host 192.168.204.113:988: is it running a compatible version of Lustre? [ 7753.074189] LNetError: Skipped 4 previous similar messages [ 7756.361536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7756.426618] Lustre: 463457:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7756.456378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7756.458914] Lustre: Skipped 1 previous similar message [ 7756.498252] Lustre: lustre-MDT0001: new disk, initializing [ 7756.511241] Lustre: 464261:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.113@tcp: can't add NID 192.168.204.113@tcp130: rc = -75 [ 7756.514226] Lustre: 464261:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7756.573567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7756.597211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7756.602720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7757.152739] LNetError: 463464:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [ 7757.162888] LNetError: 463464:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [ 7757.171114] LNetError: Protocol error connecting to 192.168.204.113@tcp126 at host 192.168.204.113:988: is it running a compatible version of Lustre? [ 7757.181477] LNetError: Skipped 1 previous similar message [ 7758.326730] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7763.373634] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7763.470123] Lustre: 464877:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.113@tcp: can't add NID 192.168.204.113@tcp130: rc = -75 [ 7763.476262] Lustre: 464877:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7765.344765] LNetError: 463462:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [ 7765.351414] LNetError: Protocol error connecting to 192.168.204.113@tcp126 at host 192.168.204.113:988: is it running a compatible version of Lustre? [ 7766.150692] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7769.607827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7769.814859] Lustre: lustre-OST0000: new disk, initializing [ 7769.818450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7769.823832] Lustre: 465433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7769.887517] Lustre: 465438:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.113@tcp: can't add NID 192.168.204.113@tcp130: rc = -75 [ 7769.893668] Lustre: 465438:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7769.926084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7772.771961] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7773.285514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7773.291875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7773.363294] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7777.880727] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7781.728845] LNetError: 463465:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [ 7781.735555] LNetError: Protocol error connecting to 192.168.204.113@tcp126 at host 192.168.204.113:988: is it running a compatible version of Lustre? [ 7783.908793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7783.913900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7783.915636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7783.921638] Lustre: Skipped 1 previous similar message [ 7787.633654] Lustre: server umount lustre-OST0000 complete [ 7792.608691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7792.613756] LustreError: Skipped 1 previous similar message [ 7792.616717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.625780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7792.629710] Lustre: Skipped 1 previous similar message [ 7794.146467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7794.146902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.706836] Lustre: server umount lustre-MDT0000 complete [ 7797.547579] LustreError: 464260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782865059 with bad export cookie 5186754994537722677 [ 7797.555106] LustreError: 464260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7797.556451] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7797.745699] Lustre: server umount lustre-MDT0001 complete [ 7798.114107] LNetError: 460904:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [ 7798.120763] LNetError: 460904:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [ 7798.125838] LNetError: Protocol error connecting to 192.168.204.113@tcp127 at host 192.168.204.113:988: is it running a compatible version of Lustre? [ 7798.132671] LNetError: Skipped 8 previous similar messages [ 7802.823893] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7804.318540] Key type lgssc unregistered [ 7804.525275] LNet: 467342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7804.530175] LNetError: 467342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7804.534501] LNet: Removed LNI 192.168.204.113@tcp [ 7805.599166] LNet: Removed LNI 192.168.204.113@tcp100 [ 7805.601897] LNetError: 467342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7805.606673] LNetError: 467342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7805.611806] LNet: Removed LNI 192.168.204.113@tcp101 [ 7806.623161] LNet: Removed LNI 192.168.204.113@tcp102 [ 7806.625844] LNetError: 467342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7806.630684] LNetError: 467342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7806.635918] LNet: Removed LNI 192.168.204.113@tcp103 [ 7807.647244] LNet: Removed LNI 192.168.204.113@tcp104 [ 7808.671173] LNet: Removed LNI 192.168.204.113@tcp105 [ 7808.673941] LNetError: 467342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7808.678816] LNetError: 467342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7808.684100] LNet: Removed LNI 192.168.204.113@tcp106 [ 7809.695170] LNet: Removed LNI 192.168.204.113@tcp107 [ 7810.719190] LNet: Removed LNI 192.168.204.113@tcp108 [ 7811.743191] LNet: Removed LNI 192.168.204.113@tcp109 [ 7812.767158] LNet: Removed LNI 192.168.204.113@tcp110 [ 7812.769839] LNetError: 467342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7812.774673] LNetError: 467342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7812.779861] LNet: Removed LNI 192.168.204.113@tcp111 [ 7813.791160] LNet: Removed LNI 192.168.204.113@tcp112 [ 7814.815122] LNet: Removed LNI 192.168.204.113@tcp113 [ 7815.839158] LNet: Removed LNI 192.168.204.113@tcp114 [ 7816.863155] LNet: Removed LNI 192.168.204.113@tcp115 [ 7817.887210] LNet: Removed LNI 192.168.204.113@tcp116 [ 7818.911158] LNet: Removed LNI 192.168.204.113@tcp117 [ 7819.935170] LNet: Removed LNI 192.168.204.113@tcp118 [ 7820.959248] LNet: Removed LNI 192.168.204.113@tcp119 [ 7820.962042] LNetError: 467342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7820.966689] LNetError: 467342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7820.971904] LNet: Removed LNI 192.168.204.113@tcp120 [ 7821.983280] LNet: Removed LNI 192.168.204.113@tcp121 [ 7823.007230] LNet: Removed LNI 192.168.204.113@tcp122 [ 7824.031185] LNet: Removed LNI 192.168.204.113@tcp123 [ 7825.055189] LNet: Removed LNI 192.168.204.113@tcp124 [ 7826.079225] LNet: Removed LNI 192.168.204.113@tcp125 [ 7826.082110] LNet: Removed LNI 192.168.204.113@tcp126 [ 7827.103210] LNet: Removed LNI 192.168.204.113@tcp127 [ 7828.127164] LNet: Removed LNI 192.168.204.113@tcp128 [ 7829.151217] LNet: Removed LNI 192.168.204.113@tcp129 [ 7829.154045] LNet: Removed LNI 192.168.204.113@tcp130 [ 7829.156666] LNet: Removed LNI 192.168.204.113@tcp131 [ 7829.159343] LNet: Removed LNI 192.168.204.113@tcp132 [ 7829.162091] LNet: Removed LNI 192.168.204.113@tcp133 [ 7829.164716] LNet: Removed LNI 192.168.204.113@tcp134 [ 7829.174375] LNet: Removed LNI 192.168.204.113@tcp135 [ 7829.609516] Key type .llcrypt unregistered [ 7829.611746] Key type ._llcrypt unregistered [ 7839.419344] Key type ._llcrypt registered [ 7839.420778] Key type .llcrypt registered [ 7839.472797] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 7846.947073] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7847.340561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7847.395306] alg: No test for adler32 (adler32-zlib) [ 7848.268626] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 7848.364851] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 7849.951219] Key type lgssc registered [ 7850.582128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7866.655374] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:18:47 (1782865127) [ 7874.964164] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7876.081926] Key type lgssc unregistered [ 7876.226780] LNet: 471401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7876.231984] LNetError: 471401:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7876.245760] LNet: Removed LNI 192.168.204.113@tcp [ 7876.677500] Key type .llcrypt unregistered [ 7876.679787] Key type ._llcrypt unregistered [ 7884.852757] Key type ._llcrypt registered [ 7884.854463] Key type .llcrypt registered [ 7884.917974] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7885.296272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7885.304094] alg: No test for adler32 (adler32-zlib) [ 7886.184081] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 7886.285825] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 7887.895149] Key type lgssc registered [ 7888.280614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7893.003160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7893.011878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7894.094199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7894.110807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7894.160503] Lustre: lustre-MDT0000: new disk, initializing [ 7894.185333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7894.193550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7896.056326] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7901.014908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7901.064260] Lustre: 472591:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7901.099191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7901.101389] Lustre: Skipped 1 previous similar message [ 7901.142817] Lustre: lustre-MDT0001: new disk, initializing [ 7901.190004] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7901.209255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7901.215142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7903.181747] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7908.213650] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7910.959894] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7914.202062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7914.377401] Lustre: lustre-OST0000: new disk, initializing [ 7914.380131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7914.384062] Lustre: 474560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7914.438151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7917.041853] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7920.657904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7920.666315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7920.730585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7921.816744] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7923.525534] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing lnet_if_list [ 7925.816235] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7925.838854] LNet: Added LNI 192.168.204.113@tcp10 [8/256/0/180] [ 7928.478146] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7928.489416] LNet: Added LNI 192.168.204.113@tcp11 [8/256/0/180] [ 7930.276420] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7930.293969] LNet: Added LNI 192.168.204.113@tcp12 [8/256/0/180] [ 7932.252512] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7932.269488] LNet: Added LNI 192.168.204.113@tcp13 [8/256/0/180] [ 7934.035183] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7934.050500] LNet: Added LNI 192.168.204.113@tcp14 [8/256/0/180] [ 7935.722775] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7935.737118] LNet: Added LNI 192.168.204.113@tcp15 [8/256/0/180] [ 7937.486497] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7937.505273] LNet: Added LNI 192.168.204.113@tcp16 [8/256/0/180] [ 7939.480552] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7939.495955] LNet: Added LNI 192.168.204.113@tcp17 [8/256/0/180] [ 7941.490507] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7941.504991] LNet: Added LNI 192.168.204.113@tcp18 [8/256/0/180] [ 7943.422357] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7943.437979] LNet: Added LNI 192.168.204.113@tcp19 [8/256/0/180] [ 7945.238583] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7945.253450] LNet: Added LNI 192.168.204.113@tcp20 [8/256/0/180] [ 7947.392294] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7947.401439] LNet: Added LNI 192.168.204.113@tcp21 [8/256/0/180] [ 7949.256952] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7949.270742] LNet: Added LNI 192.168.204.113@tcp22 [8/256/0/180] [ 7951.372326] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7951.390871] LNet: Added LNI 192.168.204.113@tcp23 [8/256/0/180] [ 7953.700975] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7953.720519] LNet: Added LNI 192.168.204.113@tcp24 [8/256/0/180] [ 7955.780740] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7955.798738] LNet: Added LNI 192.168.204.113@tcp25 [8/256/0/180] [ 7958.273775] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7958.291441] LNet: Added LNI 192.168.204.113@tcp26 [8/256/0/180] [ 7960.650531] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7960.668981] LNet: Added LNI 192.168.204.113@tcp27 [8/256/0/180] [ 7963.128473] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7963.146933] LNet: Added LNI 192.168.204.113@tcp28 [8/256/0/180] [ 7965.452480] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7965.464726] LNet: Added LNI 192.168.204.113@tcp29 [8/256/0/180] [ 7967.847171] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7967.859553] LNet: Added LNI 192.168.204.113@tcp30 [8/256/0/180] [ 7969.981358] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7969.993273] LNet: Added LNI 192.168.204.113@tcp31 [8/256/0/180] [ 7972.408283] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7972.427313] LNet: Added LNI 192.168.204.113@tcp32 [8/256/0/180] [ 7975.096991] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7975.116256] LNet: Added LNI 192.168.204.113@tcp33 [8/256/0/180] [ 7977.944711] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7977.955721] LNet: Added LNI 192.168.204.113@tcp34 [8/256/0/180] [ 7980.639923] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7980.658574] LNet: Added LNI 192.168.204.113@tcp35 [8/256/0/180] [ 7983.419188] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7983.433662] LNet: Added LNI 192.168.204.113@tcp36 [8/256/0/180] [ 7986.095137] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7986.110164] LNet: Added LNI 192.168.204.113@tcp37 [8/256/0/180] [ 7988.743112] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7988.757507] LNet: Added LNI 192.168.204.113@tcp38 [8/256/0/180] [ 7991.191503] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7991.206819] LNet: Added LNI 192.168.204.113@tcp39 [8/256/0/180] [ 7994.077882] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7994.090911] LNet: Added LNI 192.168.204.113@tcp40 [8/256/0/180] [ 7996.905988] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7996.923984] LNet: Added LNI 192.168.204.113@tcp41 [8/256/0/180] [ 8000.179305] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8000.193930] LNet: Added LNI 192.168.204.113@tcp42 [8/256/0/180] [ 8003.287944] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8003.306530] LNet: Added LNI 192.168.204.113@tcp43 [8/256/0/180] [ 8006.721001] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8006.733543] LNet: Added LNI 192.168.204.113@tcp44 [8/256/0/180] [ 8009.964055] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8009.982573] LNet: Added LNI 192.168.204.113@tcp45 [8/256/0/180] [ 8013.282244] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8013.294272] LNet: Added LNI 192.168.204.113@tcp46 [8/256/0/180] [ 8016.417968] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8016.434921] LNet: Added LNI 192.168.204.113@tcp47 [8/256/0/180] [ 8019.873248] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8019.891350] LNet: Added LNI 192.168.204.113@tcp48 [8/256/0/180] [ 8023.254609] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8023.271746] LNet: Added LNI 192.168.204.113@tcp49 [8/256/0/180] [ 8026.821990] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8026.840723] LNet: Added LNI 192.168.204.113@tcp50 [8/256/0/180] [ 8038.372760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8038.377044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8038.380556] Lustre: Skipped 2 previous similar messages [ 8038.384374] Lustre: Skipped 1 previous similar message [ 8041.010848] Lustre: server umount lustre-MDT0000 complete [ 8042.606982] LustreError: 472584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782865304 with bad export cookie 13592421416018598571 [ 8042.614188] LustreError: 472584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8042.616520] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8042.764077] Lustre: server umount lustre-MDT0001 complete [ 8051.793940] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 8055.408390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8058.847249] Lustre: 471864:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782865305/real 1782865305] req@000000003c59655b x1869469605520640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782865321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8058.862590] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8062.943285] Lustre: 471864:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782865309/real 1782865309] req@00000000df59f77c x1869469605520768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782865325 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8063.967102] Lustre: 471865:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782865310/real 1782865310] req@000000007c5b272f x1869469605520896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782865326 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8068.063542] LustreError: 471862:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003fa066d8 x1869469605521664/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8068.230993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8069.087147] Lustre: 471866:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782865315/real 1782865315] req@0000000034e0eea1 x1869469605521152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782865331 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8070.009582] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8073.721885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8073.907134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8074.911092] Lustre: 471863:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782865320/real 1782865320] req@000000000be59136 x1869469605521280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782865336 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8075.345712] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8077.577695] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8077.990668] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8079.281335] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8088.548941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.550252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8088.552106] Lustre: Skipped 1 previous similar message [ 8088.554773] Lustre: Skipped 1 previous similar message [ 8092.306298] Lustre: server umount lustre-OST0000 complete [ 8093.665275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8093.665844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8093.671577] Lustre: Skipped 1 previous similar message [ 8093.675144] Lustre: Skipped 1 previous similar message [ 8098.784066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8098.786198] Lustre: Skipped 1 previous similar message [ 8099.536876] Lustre: server umount lustre-MDT0000 complete [ 8100.606157] LustreError: 472584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782865362 with bad export cookie 13592421416018600636 [ 8100.608662] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8100.609257] LustreError: 472584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8100.714830] Lustre: server umount lustre-MDT0001 complete [ 8103.998604] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 8104.842540] Key type lgssc unregistered [ 8104.982505] LNet: 501524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8104.984922] LNetError: 501524:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8104.987078] LNet: Removed LNI 192.168.204.113@tcp [ 8104.988129] LNet: Removed LNI 192.168.204.113@tcp10 [ 8104.989281] LNet: Removed LNI 192.168.204.113@tcp11 [ 8104.990302] LNet: Removed LNI 192.168.204.113@tcp12 [ 8104.991373] LNet: Removed LNI 192.168.204.113@tcp13 [ 8104.992456] LNet: Removed LNI 192.168.204.113@tcp14 [ 8104.993456] LNet: Removed LNI 192.168.204.113@tcp15 [ 8104.994519] LNet: Removed LNI 192.168.204.113@tcp16 [ 8104.995521] LNet: Removed LNI 192.168.204.113@tcp17 [ 8104.996481] LNet: Removed LNI 192.168.204.113@tcp18 [ 8104.997521] LNet: Removed LNI 192.168.204.113@tcp19 [ 8104.998508] LNet: Removed LNI 192.168.204.113@tcp20 [ 8104.999521] LNet: Removed LNI 192.168.204.113@tcp21 [ 8105.000490] LNet: Removed LNI 192.168.204.113@tcp22 [ 8105.001519] LNet: Removed LNI 192.168.204.113@tcp23 [ 8105.002616] LNet: Removed LNI 192.168.204.113@tcp24 [ 8105.003641] LNet: Removed LNI 192.168.204.113@tcp25 [ 8105.004681] LNet: Removed LNI 192.168.204.113@tcp26 [ 8105.005720] LNet: Removed LNI 192.168.204.113@tcp27 [ 8105.006788] LNet: Removed LNI 192.168.204.113@tcp28 [ 8105.007828] LNet: Removed LNI 192.168.204.113@tcp29 [ 8105.008905] LNet: Removed LNI 192.168.204.113@tcp30 [ 8105.009933] LNet: Removed LNI 192.168.204.113@tcp31 [ 8105.010955] LNet: Removed LNI 192.168.204.113@tcp32 [ 8105.011968] LNet: Removed LNI 192.168.204.113@tcp33 [ 8105.013055] LNet: Removed LNI 192.168.204.113@tcp34 [ 8105.014065] LNet: Removed LNI 192.168.204.113@tcp35 [ 8105.015134] LNet: Removed LNI 192.168.204.113@tcp36 [ 8105.016165] LNet: Removed LNI 192.168.204.113@tcp37 [ 8105.017206] LNet: Removed LNI 192.168.204.113@tcp38 [ 8105.018267] LNet: Removed LNI 192.168.204.113@tcp39 [ 8105.019276] LNet: Removed LNI 192.168.204.113@tcp40 [ 8105.020274] LNet: Removed LNI 192.168.204.113@tcp41 [ 8105.021205] LNet: Removed LNI 192.168.204.113@tcp42 [ 8105.022211] LNet: Removed LNI 192.168.204.113@tcp43 [ 8105.023231] LNet: Removed LNI 192.168.204.113@tcp44 [ 8105.024168] LNet: Removed LNI 192.168.204.113@tcp45 [ 8105.025188] LNet: Removed LNI 192.168.204.113@tcp46 [ 8105.026230] LNet: Removed LNI 192.168.204.113@tcp47 [ 8105.027242] LNet: Removed LNI 192.168.204.113@tcp48 [ 8105.028274] LNet: Removed LNI 192.168.204.113@tcp49 [ 8105.033260] LNet: Removed LNI 192.168.204.113@tcp50 [ 8105.298813] Key type .llcrypt unregistered [ 8105.300202] Key type ._llcrypt unregistered [ 8111.639627] Key type ._llcrypt registered [ 8111.641025] Key type .llcrypt registered [ 8111.683448] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 8119.190071] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 8119.512551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8119.553626] alg: No test for adler32 (adler32-zlib) [ 8120.403407] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 8120.486833] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 8122.071165] Key type lgssc registered [ 8122.411746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8136.786439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8136.799521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8137.930600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8137.950434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8138.000073] Lustre: lustre-MDT0000: new disk, initializing [ 8138.029613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8138.036896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8139.835555] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8145.301633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8145.367832] Lustre: 504695:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8145.413619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8145.417607] Lustre: Skipped 1 previous similar message [ 8145.474687] Lustre: lustre-MDT0001: new disk, initializing [ 8145.533480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8145.554998] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8145.562813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8147.530731] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8152.666196] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8155.444315] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8159.220469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8159.335532] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8159.376346] Lustre: lustre-OST0000: new disk, initializing [ 8159.379459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8159.384262] Lustre: 506710:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8159.423737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8162.106470] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8163.852142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8163.858930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8163.912314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8166.851401] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8174.050302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8174.054669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.056583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8174.063448] Lustre: Skipped 1 previous similar message [ 8179.170614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8179.173017] Lustre: Skipped 2 previous similar messages [ 8184.287146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8184.293838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8184.298186] Lustre: Skipped 1 previous similar message [ 8184.381823] Lustre: server umount lustre-OST0000 complete [ 8186.848459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8186.853379] LustreError: Skipped 1 previous similar message [ 8186.856269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8186.864717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8189.409412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8192.235918] Lustre: server umount lustre-MDT0000 complete [ 8193.983795] LustreError: 505490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782865456 with bad export cookie 11218122234754104177 [ 8193.989166] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8193.991147] LustreError: 505490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8194.147293] Lustre: server umount lustre-MDT0001 complete [ 8198.830488] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 8199.678294] Key type lgssc unregistered [ 8199.794329] LNet: 508613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8199.796629] LNetError: 508613:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8199.804244] LNet: Removed LNI 192.168.204.113@tcp [ 8200.071886] Key type .llcrypt unregistered [ 8200.073080] Key type ._llcrypt unregistered [ 8209.175647] Key type ._llcrypt registered [ 8209.177649] Key type .llcrypt registered [ 8209.242509] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 8209.602225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8209.635489] alg: No test for adler32 (adler32-zlib) [ 8210.498623] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 8210.597077] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 8212.183160] Key type lgssc registered [ 8212.758460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8216.518748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8216.530094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8217.718849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8219.484527] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8223.559544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8223.737491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8225.512971] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8229.033235] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8231.687126] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8234.927578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8235.120309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8237.301859] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8240.747812] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8255.804587] loop0: detected capacity change from 0 to 204800000 [ 8255.891121] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8255.896200] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8255.901890] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8258.934225] loop0: detected capacity change from 0 to 204800000 [ 8259.017862] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8259.024290] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8259.030750] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8261.527759] loop0: detected capacity change from 0 to 204800000 [ 8264.079075] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8264.159306] Lustre: fs15246-OST0000: new disk, initializing [ 8264.162897] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8264.168454] Lustre: 513474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8264.201292] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8266.836647] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8271.957689] loop1: detected capacity change from 0 to 204800000 [ 8274.509587] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8274.585565] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8274.621478] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8274.675381] Lustre: fs15246-MDT0000: new disk, initializing [ 8274.746341] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8274.762602] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8274.840454] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8275.871902] Lustre: 514596:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000e8f84874 x1869469945202688/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782865554 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8276.631348] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8284.962146] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8284.968326] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8284.984936] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8289.256320] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.265417] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8293.227754] Lustre: server umount fs15246-MDT0000 complete [ 8294.372067] LustreError: 509855:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.376155] LustreError: 509855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8296.385356] Lustre: server umount fs15246-OST0000 complete [ 8301.024317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8301.029470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8301.038214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8304.614212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8304.614535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8304.626258] Lustre: Skipped 1 previous similar message [ 8306.664316] Lustre: server umount lustre-OST0000 complete [ 8309.730169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8309.731635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8309.738093] Lustre: Skipped 1 previous similar message [ 8309.744960] Lustre: Skipped 1 previous similar message [ 8314.613591] Lustre: server umount lustre-MDT0000 complete [ 8314.849388] LustreError: 509859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8314.859680] LustreError: 509859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8316.420240] LustreError: 509842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782865578 with bad export cookie 14127243725141732541 [ 8316.425992] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8316.428351] LustreError: 509842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8316.564214] Lustre: server umount lustre-MDT0001 complete [ 8320.759630] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 8322.240330] Key type lgssc unregistered [ 8322.400618] LNet: 517685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8322.405599] LNetError: 517685:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8322.417541] LNet: Removed LNI 192.168.204.113@tcp [ 8322.880072] Key type .llcrypt unregistered [ 8322.882269] Key type ._llcrypt unregistered [ 8331.926066] Key type ._llcrypt registered [ 8331.927796] Key type .llcrypt registered [ 8331.985552] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 8332.401195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8332.410433] alg: No test for adler32 (adler32-zlib) [ 8333.270470] Lustre: Lustre: Build Version: 2.17.54_83_g326f97e [ 8333.374845] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 8334.959223] Key type lgssc registered [ 8335.537630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8352.904354] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 8357.043211] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 8374.092638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8374.105358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.228837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8375.248086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8375.305458] Lustre: lustre-MDT0000: new disk, initializing [ 8375.339748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8375.351450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8377.037223] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8382.685239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8382.759582] Lustre: 522122:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8382.803857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8382.808220] Lustre: Skipped 1 previous similar message [ 8382.857568] Lustre: lustre-MDT0001: new disk, initializing [ 8382.899506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8382.920397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8382.926608] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8384.734820] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8389.697323] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8392.213474] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8395.524974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8395.678068] Lustre: lustre-OST0000: new disk, initializing [ 8395.681456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8395.686656] Lustre: 524090:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8395.737227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8398.305834] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8403.208998] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8403.471657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8403.480937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8403.530781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8405.254303] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8405.343817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8407.131881] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8407.257762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8408.544943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8408.547153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8408.549144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8408.550904] LustreError: Skipped 1 previous similar message [ 8408.556443] Lustre: Skipped 1 previous similar message [ 8410.281884] Lustre: server umount lustre-OST0000 complete [ 8413.666571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8413.667555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8413.668747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8413.668756] Lustre: Skipped 1 previous similar message [ 8413.688083] Lustre: Skipped 1 previous similar message [ 8418.229729] Lustre: server umount lustre-MDT0000 complete [ 8420.071239] LustreError: 522114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782865682 with bad export cookie 589562167516854899 [ 8420.076989] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8420.080806] LustreError: 522114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8420.222536] Lustre: server umount lustre-MDT0001 complete [ 8430.662243] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 8435.521338] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 8451.247749] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8312 sec ========== 20:28:32 (1782865712) [ 8452.124331] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:28:33 (1782865713) === [ 8453.859862] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:28:35 (1782865715) === [ 8473.017265] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 8474.537326] Key type lgssc unregistered [ 8474.693654] LNet: 530721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8474.698598] LNetError: 530721:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8474.707523] LNet: Removed LNI 192.168.204.113@tcp [ 8475.174435] Key type .llcrypt unregistered [ 8475.176822] Key type ._llcrypt unregistered