[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 497384886 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003203] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008576] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011014] pid_max: default: 32768 minimum: 301 [ 0.012144] LSM: Security Framework initializing [ 0.013000] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014069] *** VALIDATE selinux *** [ 0.023504] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029499] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034144] *** VALIDATE tmpfs *** [ 0.035470] *** VALIDATE proc *** [ 0.036261] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.039123] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042034] Spectre V2 : User space: Vulnerable [ 0.043010] Speculative Store Bypass: Vulnerable [ 0.046669] debug: unmapping init [mem 0xffffffff93e59000-0xffffffff93e60fff] [ 0.048874] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049757] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050027] ... version: 2 [ 0.051014] ... bit width: 48 [ 0.052020] ... generic registers: 4 [ 0.053014] ... value mask: 0000ffffffffffff [ 0.054018] ... max period: 00007fffffffffff [ 0.055020] ... fixed-purpose events: 3 [ 0.056011] ... event mask: 000000070000000f [ 0.057365] rcu: Hierarchical SRCU implementation. [ 0.059531] smp: Bringing up secondary CPUs ... [ 0.060637] x86: Booting SMP configuration: [ 0.061028] .... node #0, CPUs: #1 #2 #3 [ 0.065078] smp: Brought up 1 node, 4 CPUs [ 0.067015] smpboot: Max logical packages: 1 [ 0.068020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144043] node 0 deferred pages initialised in 74ms [ 0.148371] devtmpfs: initialized [ 0.150414] x86/mm: Memory block size: 128MB [ 0.154774] gcov: version magic: 0x41383552 [ 0.159362] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.164218] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.167512] pinctrl core: initialized pinctrl subsystem [ 0.170290] [ 0.170975] ************************************************************* [ 0.173025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176023] ** ** [ 0.179024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.181020] ** ** [ 0.184022] ** This means that this kernel is built to expose internal ** [ 0.186021] ** IOMMU data structures, which may compromise security on ** [ 0.189026] ** your system. ** [ 0.193023] ** ** [ 0.195015] ** If you see this message and you are not debugging the ** [ 0.198023] ** kernel, report this immediately to your vendor! ** [ 0.201022] ** ** [ 0.203021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.206025] ************************************************************* [ 0.209855] NET: Registered protocol family 16 [ 0.212569] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.216127] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.218067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.222098] cpuidle: using governor menu [ 0.223677] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.226589] PCI: Using configuration type 1 for base access [ 0.228146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.237179] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.238039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.240096] cryptd: max_cpu_qlen set to 1000 [ 0.243325] ACPI: Added _OSI(Module Device) [ 0.245048] ACPI: Added _OSI(Processor Device) [ 0.247032] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.249018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.255408] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.262614] ACPI: Interpreter enabled [ 0.265078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.266019] ACPI: Using IOAPIC for interrupt routing [ 0.269133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.273523] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.285000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.287073] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.290031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.294190] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.300569] acpiphp: Slot [2] registered [ 0.302208] acpiphp: Slot [5] registered [ 0.303200] acpiphp: Slot [6] registered [ 0.305161] acpiphp: Slot [7] registered [ 0.306204] acpiphp: Slot [8] registered [ 0.308151] acpiphp: Slot [9] registered [ 0.310205] acpiphp: Slot [10] registered [ 0.313200] acpiphp: Slot [3] registered [ 0.314133] acpiphp: Slot [4] registered [ 0.316155] acpiphp: Slot [11] registered [ 0.317138] acpiphp: Slot [12] registered [ 0.319135] acpiphp: Slot [13] registered [ 0.320168] acpiphp: Slot [14] registered [ 0.323168] acpiphp: Slot [15] registered [ 0.324142] acpiphp: Slot [16] registered [ 0.326100] acpiphp: Slot [17] registered [ 0.328115] acpiphp: Slot [18] registered [ 0.329132] acpiphp: Slot [19] registered [ 0.331182] acpiphp: Slot [20] registered [ 0.332153] acpiphp: Slot [21] registered [ 0.334121] acpiphp: Slot [22] registered [ 0.336124] acpiphp: Slot [23] registered [ 0.337113] acpiphp: Slot [24] registered [ 0.339123] acpiphp: Slot [25] registered [ 0.340134] acpiphp: Slot [26] registered [ 0.342209] acpiphp: Slot [27] registered [ 0.344182] acpiphp: Slot [28] registered [ 0.346159] acpiphp: Slot [29] registered [ 0.348180] acpiphp: Slot [30] registered [ 0.350159] acpiphp: Slot [31] registered [ 0.352087] PCI host bridge to bus 0000:00 [ 0.353031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.355028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.358032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.360039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.363037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.367040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.369263] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.373219] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.376684] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.386828] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.393814] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.396026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.398022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.400026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.403614] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.407153] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.410066] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.413980] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.420028] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.435022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.440020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.447473] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.453022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.461024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.480025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.491240] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.498027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.506018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.524018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.533357] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.541025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.547024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.563045] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.577985] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.586021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.593020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.609030] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.620000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.626026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.635019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.654026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.665437] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.674027] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.680024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.697025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.712274] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.714406] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.717437] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.721509] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.723265] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.730043] iommu: Default domain type: Passthrough [ 0.731600] SCSI subsystem initialized [ 0.732243] ACPI: bus type USB registered [ 0.733170] usbcore: registered new interface driver usbfs [ 0.735123] usbcore: registered new interface driver hub [ 0.737154] usbcore: registered new device driver usb [ 0.740221] pps_core: LinuxPPS API ver. 1 registered [ 0.742013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.745144] PTP clock support registered [ 0.747187] EDAC MC: Ver: 3.0.0 [ 0.748796] PCI: Using ACPI for IRQ routing [ 0.749851] NetLabel: Initializing [ 0.751016] NetLabel: domain hash size = 128 [ 0.752011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.754257] NetLabel: unlabeled traffic allowed by default [ 0.757593] vgaarb: loaded [ 0.759404] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.760021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.770465] clocksource: Switched to clocksource kvm-clock [ 0.888401] VFS: Disk quotas dquot_6.6.0 [ 0.890323] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.893211] *** VALIDATE ramfs *** [ 0.894670] *** VALIDATE hugetlbfs *** [ 0.896497] pnp: PnP ACPI init [ 0.899359] pnp: PnP ACPI: found 6 devices [ 0.917401] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.921590] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.923673] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.926045] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.928991] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.931675] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.934801] NET: Registered protocol family 2 [ 0.937523] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.942625] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.946219] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.954534] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.958039] TCP: Hash tables configured (established 65536 bind 65536) [ 0.961361] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.965098] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.968509] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971943] NET: Registered protocol family 1 [ 0.975148] RPC: Registered named UNIX socket transport module. [ 0.978632] RPC: Registered udp transport module. [ 0.980817] RPC: Registered tcp transport module. [ 0.982971] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.985866] NET: Registered protocol family 44 [ 0.987640] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.989941] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.992253] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.994715] PCI: CLS 0 bytes, default 64 [ 0.997435] Unpacking initramfs... [ 2.429328] debug: unmapping init [mem 0xffff9399bcc54000-0xffff9399bffbffff] [ 2.433741] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.435579] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.438623] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.932041] Initialise system trusted keyrings [ 2.933969] Key type blacklist registered [ 2.936088] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.947319] zbud: loaded [ 2.950531] *** VALIDATE nfs *** [ 2.952064] *** VALIDATE nfs4 *** [ 2.953873] pstore: using deflate compression [ 2.973937] Platform Keyring initialized [ 3.086188] NET: Registered protocol family 38 [ 3.088059] Key type asymmetric registered [ 3.089729] Asymmetric key parser 'x509' registered [ 3.091984] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.095297] io scheduler mq-deadline registered [ 3.097282] io scheduler kyber registered [ 3.099216] io scheduler bfq registered [ 3.101499] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.104781] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.107559] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.110718] ACPI: Power Button [PWRF] [ 3.115828] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.123046] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.138133] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.145250] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.169134] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.197175] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.225054] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.230801] Non-volatile memory driver v1.3 [ 3.232435] Linux agpgart interface v0.103 [ 3.271385] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.274518] vda: detected capacity change from 0 to 74760192 [ 3.295495] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.298977] vdb: detected capacity change from 0 to 1073741824 [ 3.315769] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.317915] vdc: detected capacity change from 0 to 2621440000 [ 3.331792] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.335033] vdd: detected capacity change from 0 to 2621440000 [ 3.352723] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.356038] vde: detected capacity change from 0 to 4294967296 [ 3.376331] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.379517] vdf: detected capacity change from 0 to 4294967296 [ 3.388695] libphy: Fixed MDIO Bus: probed [ 3.397579] usbcore: registered new interface driver usbserial_generic [ 3.401167] usbserial: USB Serial support registered for generic [ 3.404697] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.411512] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.413242] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.416314] mousedev: PS/2 mouse device common for all mice [ 3.419805] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.420944] rtc_cmos 00:05: RTC can wake from S4 [ 3.427146] rtc_cmos 00:05: registered as rtc0 [ 3.427398] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.429110] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.435302] intel_pstate: CPU model not supported [ 3.438046] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.440374] hid: raw HID events driver (C) Jiri Kosina [ 3.444466] usbcore: registered new interface driver usbhid [ 3.446493] usbhid: USB HID core driver [ 3.448153] drop_monitor: Initializing network drop monitor service [ 3.450531] Initializing XFRM netlink socket [ 3.452543] NET: Registered protocol family 10 [ 3.455872] Segment Routing with IPv6 [ 3.457352] NET: Registered protocol family 17 [ 3.459435] mpls_gso: MPLS GSO support [ 3.466531] RAS: Correctable Errors collector initialized. [ 3.469039] AVX version of gcm_enc/dec engaged. [ 3.470869] AES CTR mode by8 optimization enabled [ 3.550976] sched_clock: Marking stable (3550939397, 0)->(4512897265, -961957868) [ 3.554557] registered taskstats version 1 [ 3.557349] Loading compiled-in X.509 certificates [ 3.559316] zswap: loaded using pool lzo/zbud [ 3.586953] Key type big_key registered [ 3.600452] Key type encrypted registered [ 3.602028] ima: No TPM chip found, activating TPM-bypass! [ 3.604066] ima: Allocated hash algorithm: sha1 [ 3.605644] ima: No architecture policies found [ 3.607369] evm: Initialising EVM extended attributes: [ 3.609223] evm: security.selinux [ 3.610342] evm: security.ima [ 3.611442] evm: security.capability [ 3.612508] evm: HMAC attrs: 0x1 [ 3.614854] rtc_cmos 00:05: setting system clock to 2026-09-01 23:36:53 UTC (1788305813) [ 3.621135] debug: unmapping init [mem 0xffffffff94e03000-0xffffffff94ffffff] [ 3.624427] debug: unmapping init [mem 0xffffffff93b82000-0xffffffff93e58fff] [ 3.634770] Write protecting the kernel read-only data: 28672k [ 3.638420] debug: unmapping init [mem 0xffffffff92203000-0xffffffff923fffff] [ 3.641248] debug: unmapping init [mem 0xffffffff92b14000-0xffffffff92bfffff] [ 3.675723] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.684441] systemd[1]: Detected virtualization kvm. [ 3.686479] systemd[1]: Detected architecture x86-64. [ 3.688277] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.717128] systemd[1]: No hostname configured. [ 3.718759] systemd[1]: Set hostname to . [ 3.720557] random: systemd: uninitialized urandom read (16 bytes read) [ 3.722980] systemd[1]: Initializing machine ID from random generator. [ 3.850483] random: systemd: uninitialized urandom read (16 bytes read) [ 3.852756] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.858078] random: systemd: uninitialized urandom read (16 bytes read) [ 3.861399] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.866547] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.550995] device-mapper: uevent: version 1.0.3 [ 4.555194] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.337510] virtio_net virtio0 ens2: renamed from eth0 [ 5.352440] random: fast init done [ 5.394772] scsi host0: ata_piix [ 5.405988] scsi host1: ata_piix [ 5.409808] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.413064] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.807646] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.116735] random: crng init done [ 10.118129] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.542639] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.673354] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.959834] SELinux: Disabled at runtime. [ 12.022942] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.031418] systemd[1]: Detected virtualization kvm. [ 12.033525] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.578307] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.581810] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.586558] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.592366] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.596178] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.602691] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.610831] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 12.693927] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.010402] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.312424] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.388626] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.461092] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.482769] EDAC sbridge: Ver: 1.1.2 [ 15.007882] Key type dns_resolver registered [ 15.325748] NFS: Registering the id_resolver key type [ 15.328114] Key type id_resolver registered [ 15.329755] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg436-server login: [ 43.096611] libcfs: loading out-of-tree module taints kernel. [ 43.162503] Key type ._llcrypt registered [ 43.164388] Key type .llcrypt registered [ 43.275906] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 59.675540] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 60.473006] hrtimer: interrupt took 6006531 ns [ 61.492789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.523178] alg: No test for adler32 (adler32-zlib) [ 63.162559] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 64.113114] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 65.896225] Key type lgssc registered [ 67.495409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.154349] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.661254] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 136.161579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 136.183229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 137.434270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 137.460421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 137.520179] Lustre: lustre-MDT0000: new disk, initializing [ 137.590460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 137.605369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 141.834190] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 155.578641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.679268] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 155.711474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 155.715782] Lustre: Skipped 1 previous similar message [ 155.789179] Lustre: lustre-MDT0001: new disk, initializing [ 155.862667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 155.896530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 155.905657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 160.698505] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.208667] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 174.442294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 174.752321] Lustre: lustre-OST0000: new disk, initializing [ 174.758101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 174.772492] Lustre: 8441:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 174.865242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.746946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.766698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 177.843358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 180.575325] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.497541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.641287] Lustre: lustre-OST0001: new disk, initializing [ 194.644695] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 194.649884] Lustre: 9514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.717860] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.409777] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.820149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 204.829945] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 204.929482] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 213.277524] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 220.718486] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 226.812621] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing check_logdir /tmp/testlogs/ [ 232.197039] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing yml_node [ 237.101604] Lustre: DEBUG MARKER: Client: 2.17.57.44 [ 239.878680] Lustre: DEBUG MARKER: MDS: 2.17.57.44 [ 242.065662] Lustre: DEBUG MARKER: OSS: 2.17.57.44 [ 243.714745] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 1 19:40:51 EDT 2026 [ 261.856488] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 263.485405] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 279.013097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 279.028840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.047526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 281.574230] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 281.575096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 281.588832] Lustre: Skipped 2 previous similar messages [ 281.593790] Lustre: Skipped 2 previous similar messages [ 284.752740] Lustre: server umount lustre-MDT0000 complete [ 288.065323] LustreError: 7453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788306097 with bad export cookie 18180051266935450359 [ 288.072220] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 288.084631] LustreError: 7453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 291.810825] LustreError: 6512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.813673] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.831142] LustreError: 6512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 291.854912] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 291.861258] Lustre: Skipped 2 previous similar messages [ 293.862612] LustreError: 10503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.865039] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 293.893590] LustreError: 10503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 293.908275] Lustre: Skipped 1 previous similar message [ 294.425834] Lustre: server umount lustre-MDT0001 complete [ 298.319296] Lustre: server umount lustre-OST0000 complete [ 301.654577] Lustre: server umount lustre-OST0001 complete [ 307.879920] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 316.999249] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 349.105829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 349.291967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 349.315317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 349.376502] Lustre: lustre-MDT0000: new disk, initializing [ 349.470108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.482165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 352.636871] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 365.152907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 365.299482] Lustre: 15062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 365.372874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 365.384384] Lustre: Skipped 1 previous similar message [ 365.473787] Lustre: lustre-MDT0001: new disk, initializing [ 365.578556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 365.612392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 365.633844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 372.725069] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 385.066481] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 391.069115] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 398.745623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 399.189158] Lustre: lustre-OST0000: new disk, initializing [ 399.192620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 399.205422] Lustre: 17034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 399.308295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.239767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 401.247345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 401.342841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 405.597168] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 414.120410] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 418.314451] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 418.477724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 422.726703] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 422.953577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 426.981850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 426.982467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 426.990379] LustreError: Skipped 1 previous similar message [ 427.006384] Lustre: Skipped 1 previous similar message [ 427.011286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 437.221646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 437.230437] Lustre: Skipped 4 previous similar messages [ 439.264173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 439.399134] Lustre: server umount lustre-OST0000 complete [ 447.459911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 447.464593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 457.185860] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 457.417847] Lustre: server umount lustre-MDT0000 complete [ 457.700481] LustreError: 16100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 457.717015] LustreError: 16100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 460.637142] LustreError: 15054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788306270 with bad export cookie 18180051266935451885 [ 460.642971] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 460.645240] LustreError: 15054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 460.963682] Lustre: server umount lustre-MDT0001 complete [ 465.011396] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:44:33 (1788306273) [ 467.052959] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 468.885608] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:44:36 (1788306276) [ 475.129889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.585929] LustreError: 19445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.664642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 480.052250] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 480.744601] LustreError: 19446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.539503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.475690] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 502.262460] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 507.592991] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 513.630622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 520.348700] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 529.358626] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 540.648610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 540.655357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.669029] Lustre: Skipped 1 previous similar message [ 540.673987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.681267] Lustre: Skipped 4 previous similar messages [ 542.876635] Lustre: server umount lustre-OST0000 complete [ 551.397024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 551.406856] LustreError: Skipped 1 previous similar message [ 553.340427] Lustre: server umount lustre-MDT0000 complete [ 556.235081] LustreError: 19427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788306366 with bad export cookie 18180051266935452774 [ 556.236056] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.246856] LustreError: 19427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 556.516991] Lustre: server umount lustre-MDT0001 complete [ 569.090836] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 572.664926] Key type lgssc unregistered [ 572.981897] LNet: 22980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.989275] LNetError: 22980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.005872] LNet: Removed LNI 192.168.204.136@tcp [ 574.013215] Key type .llcrypt unregistered [ 574.018361] Key type ._llcrypt unregistered [ 598.577302] Key type ._llcrypt registered [ 598.579802] Key type .llcrypt registered [ 598.679661] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 600.102661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 600.160695] alg: No test for adler32 (adler32-zlib) [ 601.352486] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 601.632373] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 603.336201] Key type lgssc registered [ 604.721522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 613.291120] loop: module loaded [ 613.396268] loop0: detected capacity change from 0 to 204800000 [ 613.677155] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 613.694157] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.055937] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 620.373245] loop0: detected capacity change from 0 to 204800000 [ 620.615141] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 620.630345] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 620.642898] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 626.219829] loop0: detected capacity change from 0 to 204800000 [ 631.548919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 631.576984] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 632.818584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 632.888240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 632.965851] Lustre: lustre-MDT0000: new disk, initializing [ 633.043419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 633.071808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 637.049658] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 645.384163] loop1: detected capacity change from 0 to 204800000 [ 651.287365] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 651.682129] Lustre: lustre-OST07c6: new disk, initializing [ 651.690681] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 651.701154] Lustre: Skipped 1 previous similar message [ 651.707931] Lustre: 26178:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 651.811134] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 657.930717] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 658.021258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 658.033387] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 658.069663] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 662.189469] Lustre: 25163:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 662.197814] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 662.203139] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 662.210356] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 662.217878] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 662.225280] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 677.724775] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 677.738546] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 677.747990] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 677.757084] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 677.765633] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 677.778635] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 682.519683] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 682.530274] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 682.538416] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 682.542916] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.548256] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 682.553403] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.559880] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 682.566098] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.570752] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 682.578734] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.585409] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 682.595218] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.756843] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 685.778796] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 685.784989] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 685.790122] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.801238] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 685.814689] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.830590] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 685.840367] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.861082] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 685.874258] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.892698] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 685.906702] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.662396] Lustre: 25163:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 690.668688] Lustre: 25163:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 690.683392] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 690.691435] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.706030] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 690.716533] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.724704] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 690.731277] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.741671] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 690.748244] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.763315] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 690.772345] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.032375] Lustre: 25163:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 702.050041] Lustre: 25163:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 702.063373] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 702.073204] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 702.078911] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 702.086127] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 702.097061] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 702.106184] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 702.112308] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 702.119736] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 702.131478] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 702.150343] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 750.085320] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 750.113216] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.143580] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 750.592893] Lustre: server umount lustre-OST07c6 complete [ 758.828814] Lustre: server umount lustre-MDT0000 complete [ 776.303856] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 778.730100] Key type lgssc unregistered [ 778.982531] LNet: 28739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.992247] LNetError: 28739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.004923] LNet: Removed LNI 192.168.204.136@tcp [ 779.941161] Key type .llcrypt unregistered [ 779.945638] Key type ._llcrypt unregistered [ 797.711263] Key type ._llcrypt registered [ 797.717394] Key type .llcrypt registered [ 797.822537] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 798.775189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 798.866637] alg: No test for adler32 (adler32-zlib) [ 800.009427] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 800.331664] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 802.048168] Key type lgssc registered [ 803.311339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 810.881684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 810.894736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.380931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 816.924516] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 822.758514] LustreError: 30002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 822.782807] LustreError: 30002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 824.621373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.012805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 829.316535] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 837.885492] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 844.280915] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 851.229295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 851.585272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.710486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 857.723566] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 865.849815] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 883.682912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 883.689540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.704903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.732629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.735970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.729467] Lustre: server umount lustre-OST0000 complete [ 895.971993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 895.982024] Lustre: Skipped 1 previous similar message [ 895.986700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 895.990353] Lustre: Skipped 2 previous similar messages [ 899.475670] Lustre: server umount lustre-MDT0000 complete [ 901.091149] LustreError: 29999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 901.104469] LustreError: 29999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 902.838979] LustreError: 29985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788306712 with bad export cookie 17853226500580464935 [ 902.844621] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 902.855454] LustreError: 29985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 903.204481] Lustre: server umount lustre-MDT0001 complete [ 913.879437] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 916.020553] Key type lgssc unregistered [ 916.295689] LNet: 33743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.306889] LNetError: 33743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.323976] LNet: Removed LNI 192.168.204.136@tcp [ 917.017654] Key type .llcrypt unregistered [ 917.020937] Key type ._llcrypt unregistered [ 935.036828] Key type ._llcrypt registered [ 935.038894] Key type .llcrypt registered [ 935.122342] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 936.512167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 936.637749] alg: No test for adler32 (adler32-zlib) [ 937.712972] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 937.967933] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 939.585404] Key type lgssc registered [ 941.241107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 949.988058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 950.023933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.567268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 956.838424] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 962.017783] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 962.037365] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 965.611042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 966.251081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 971.844358] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 980.045896] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 986.751816] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 993.800609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 994.107416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.148298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1000.800610] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1008.743654] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1027.046929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.053648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1027.054991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1027.073106] Lustre: Skipped 1 previous similar message [ 1027.102274] Lustre: Skipped 1 previous similar message [ 1030.278179] Lustre: server umount lustre-OST0000 complete [ 1037.286700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.304662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1037.307480] Lustre: Skipped 1 previous similar message [ 1040.362034] Lustre: server umount lustre-MDT0000 complete [ 1042.403508] LustreError: 35014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.419476] LustreError: 35014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1043.670804] LustreError: 34994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788306853 with bad export cookie 18086638029778106894 [ 1043.679215] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1043.681987] LustreError: 34994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1044.137315] Lustre: server umount lustre-MDT0001 complete [ 1055.217820] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1057.619953] Key type lgssc unregistered [ 1057.913661] LNet: 38572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.918845] LNetError: 38572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.950395] LNet: Removed LNI 192.168.204.136@tcp [ 1058.705404] Key type .llcrypt unregistered [ 1058.707756] Key type ._llcrypt unregistered [ 1081.953245] Key type ._llcrypt registered [ 1081.955324] Key type .llcrypt registered [ 1082.063820] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1082.869238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1082.918631] alg: No test for adler32 (adler32-zlib) [ 1083.859411] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 1084.084591] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1085.768158] Key type lgssc registered [ 1086.639797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1095.171521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1095.193254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.720658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.093253] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1106.915574] LustreError: 39998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.945934] LustreError: 39998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1110.944769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.263096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1115.634788] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1123.383414] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1129.166742] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1135.266600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1135.591543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1141.384257] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1141.768534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1148.961926] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1157.092204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.104191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1157.110901] Lustre: Skipped 2 previous similar messages [ 1157.125807] Lustre: Skipped 1 previous similar message [ 1161.960801] Lustre: server umount lustre-MDT0000 complete [ 1162.219436] LustreError: 39992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.235095] LustreError: 39992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1165.501704] LustreError: 39979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788306975 with bad export cookie 15486502156960820584 [ 1165.507858] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1165.519736] LustreError: 39979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1167.329783] LustreError: 41908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.330908] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.348168] LustreError: 41908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1167.376786] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1167.380274] Lustre: Skipped 1 previous similar message [ 1170.342263] LustreError: 41908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.357302] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1170.374459] LustreError: 41908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1170.386956] Lustre: Skipped 1 previous similar message [ 1171.945740] Lustre: server umount lustre-MDT0001 complete [ 1201.120519] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1201.249165] Lustre: server umount lustre-OST0000 complete [ 1215.439176] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1217.673268] Key type lgssc unregistered [ 1217.921460] LNet: 43862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1217.926796] LNetError: 43862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.985606] LNet: Removed LNI 192.168.204.136@tcp [ 1219.898377] Key type .llcrypt unregistered [ 1219.903929] Key type ._llcrypt unregistered [ 1241.948438] Key type ._llcrypt registered [ 1241.952278] Key type .llcrypt registered [ 1242.086927] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1242.859295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1242.971252] alg: No test for adler32 (adler32-zlib) [ 1244.064195] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 1244.294544] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1246.000203] Key type lgssc registered [ 1247.332503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1255.552971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1255.571531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.066120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1261.224295] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1266.661317] LustreError: 45280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.673888] LustreError: 45280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1268.432909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.768308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1272.183849] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1279.269591] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1284.508704] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1291.699377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1292.206109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1298.171785] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1301.432886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1305.986139] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1313.762356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1313.766087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.781284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1316.855848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1316.868860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1316.870418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.877175] Lustre: Skipped 1 previous similar message [ 1318.640657] Lustre: server umount lustre-OST0000 complete [ 1340.386193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1340.396344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.411918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1340.423338] Lustre: Skipped 1 previous similar message [ 1343.431862] Lustre: server umount lustre-MDT0000 complete [ 1346.543104] LustreError: 46008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.572935] LustreError: 46008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1347.353370] LustreError: 45259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788307157 with bad export cookie 9112099880560887409 [ 1347.368871] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1347.370504] LustreError: 45259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1353.757297] Lustre: server umount lustre-MDT0001 complete [ 1366.537720] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1369.654545] Key type lgssc unregistered [ 1369.974418] LNet: 48990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.981466] LNetError: 48990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.012898] LNet: Removed LNI 192.168.204.136@tcp [ 1371.037438] Key type .llcrypt unregistered [ 1371.040419] Key type ._llcrypt unregistered [ 1402.855078] Key type ._llcrypt registered [ 1402.857184] Key type .llcrypt registered [ 1402.962103] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1403.537161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1403.559826] alg: No test for adler32 (adler32-zlib) [ 1404.519822] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 1404.705396] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1406.352192] Key type lgssc registered [ 1408.273608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1415.994778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1416.020593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.402840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.433132] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1427.946886] LustreError: 50409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.969410] LustreError: 50409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1431.838896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1432.246210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1436.976765] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1444.970665] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1451.233245] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1459.373881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1459.742214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1462.831086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1466.324313] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1476.000574] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1483.242209] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1487.285224] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1499.254233] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0120h01m46s [ 1500.828699] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1502.697695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1502.708782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1502.711376] Lustre: Skipped 1 previous similar message [ 1503.778666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1503.787614] Lustre: Skipped 4 previous similar messages [ 1505.347377] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 1505.359679] Lustre: Skipped 1 previous similar message [ 1508.950977] Lustre: server umount lustre-MDT0000 complete [ 1509.860577] LustreError: 50404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.880424] LustreError: 50404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1512.875674] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1516.013981] LustreError: 50409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1516.041833] LustreError: 50409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1520.422773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.752283] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1521.058796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.132737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1522.104317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1526.689795] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1530.547472] Lustre: DEBUG MARKER: Wait for df (36721) ... [ 1532.892383] Lustre: DEBUG MARKER: done [ 1538.529503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1538.541258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1538.573422] Lustre: Skipped 1 previous similar message [ 1538.582814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1541.603045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1541.618285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1542.630488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1542.638571] Lustre: Skipped 1 previous similar message [ 1544.393737] Lustre: server umount lustre-OST0000 complete [ 1552.866040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1552.870048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1552.898465] Lustre: Skipped 1 previous similar message [ 1552.914908] Lustre: Skipped 1 previous similar message [ 1554.225991] Lustre: server umount lustre-MDT0000 complete [ 1558.480232] LustreError: 53049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788307368 with bad export cookie 1921371911486877688 [ 1558.490046] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1558.501594] LustreError: 53049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1558.860824] Lustre: server umount lustre-MDT0001 complete [ 1570.679798] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1573.693263] Key type lgssc unregistered [ 1574.063606] LNet: 55349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.070763] LNetError: 55349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1574.085823] LNet: Removed LNI 192.168.204.136@tcp [ 1574.955726] Key type .llcrypt unregistered [ 1574.958244] Key type ._llcrypt unregistered [ 1619.531116] Key type ._llcrypt registered [ 1619.534813] Key type .llcrypt registered [ 1619.749490] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1621.378504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1621.473303] alg: No test for adler32 (adler32-zlib) [ 1622.610699] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 1623.067821] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1624.852260] Key type lgssc registered [ 1626.359286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1636.599495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1636.622619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1637.807974] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1637.826519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1637.953815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.500576] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1651.391142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1651.612426] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1651.687289] Lustre: 57559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1651.881668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1656.361609] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1665.405355] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1671.484274] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1678.712386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1678.951464] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1679.200241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1684.674642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1685.412938] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1692.756553] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1700.055574] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0120h05m07s [ 1701.593569] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1708.422994] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1711.131907] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1711.174087] LustreError: 57564:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1731.628971] Lustre: lustre-MDT0000: Client 724dd252-f2ad-43b0-99a1-6a6f8e3d9c7f (at 192.168.204.36@tcp) reconnecting [ 1751.273412] LustreError: 57564:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1754.289752] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1757.351528] Lustre: DEBUG MARKER: done [ 1766.371379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1766.380357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.413569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1766.887139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1766.893566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.919404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1771.052978] Lustre: server umount lustre-OST0000 complete [ 1777.125773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.127783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1777.138066] Lustre: Skipped 1 previous similar message [ 1781.457432] Lustre: server umount lustre-MDT0000 complete [ 1785.018402] LustreError: 59420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788307594 with bad export cookie 217411015870007115 [ 1785.021326] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1785.034341] LustreError: 59420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1785.409036] Lustre: server umount lustre-MDT0001 complete [ 1796.229360] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 1798.974668] Key type lgssc unregistered [ 1799.256359] LNet: 61889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.268223] LNetError: 61889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.296249] LNet: Removed LNI 192.168.204.136@tcp [ 1801.109439] Key type .llcrypt unregistered [ 1801.110861] Key type ._llcrypt unregistered [ 1851.403605] Key type ._llcrypt registered [ 1851.407329] Key type .llcrypt registered [ 1851.502830] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 1852.612871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1852.671846] alg: No test for adler32 (adler32-zlib) [ 1853.702366] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 1853.925064] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 1855.608328] Key type lgssc registered [ 1856.747470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1859.310475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1859.346875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.715052] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1860.753580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1860.987185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.861393] Lustre: Failing over lustre-MDT0000 [ 1862.174843] Lustre: server umount lustre-MDT0000 complete [ 1871.857910] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:07:58 (1788307678) [ 1879.012700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.675861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1885.927122] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1894.625937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.769677] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1894.811606] Lustre: 64688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1895.092312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1899.226960] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1908.267792] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1914.201814] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1921.341477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1921.495821] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1921.684158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.624562] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1929.333595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1933.899866] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1939.908917] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1945.060594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.062758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.070498] Lustre: Skipped 1 previous similar message [ 1946.524643] Lustre: server umount lustre-MDT0000 complete [ 1950.180527] LustreError: 67255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.208789] LustreError: 67255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1950.648697] LustreError: 64680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788307760 with bad export cookie 13564733407602846812 [ 1950.663487] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1950.680520] LustreError: 64680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1951.068749] Lustre: server umount lustre-MDT0001 complete [ 1955.430901] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1960.683943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1971.084770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.684586] Lustre: 63613:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788307765/real 1788307765] req@00000000b84d43ad x1875176496076416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788307781 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1971.705900] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.715509] Lustre: Skipped 1 previous similar message [ 1976.075185] LustreError: 66537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1976.161758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1976.255992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1977.121166] Lustre: 63614:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788307770/real 1788307770] req@0000000074708530 x1875176496076672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788307786 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1980.194205] LustreError: 66538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.546371] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1981.218285] Lustre: 63611:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788307775/real 1788307775] req@0000000089c36366 x1875176496076800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788307791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1986.533255] Lustre: 63611:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788307780/real 1788307780] req@00000000576985be x1875176496076928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788307796 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1986.544198] LustreError: 68461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1986.606101] LustreError: 68461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1989.325488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.734160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1994.056275] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 1995.247248] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2000.422459] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2001.538331] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2007.581451] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2015.726599] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.736409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.737192] Lustre: Skipped 1 previous similar message [ 2015.750278] Lustre: Skipped 3 previous similar messages [ 2020.325685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2020.331589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.341421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.572943] Lustre: server umount lustre-MDT0000 complete [ 2025.689056] LustreError: 66543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788307835 with bad export cookie 13564733407602849087 [ 2025.696118] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.708199] LustreError: 66543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2025.960198] LustreError: 68462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2025.965896] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.215357] Lustre: server umount lustre-MDT0001 complete [ 2033.905786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2043.644168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.815139] LustreError: 69196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.844969] LustreError: 69196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2050.920954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.033209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2055.660750] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2056.043756] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2066.124162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2071.468581] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2071.541383] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2079.562635] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2086.171809] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2097.127049] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.132987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.133625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2097.152317] Lustre: Skipped 2 previous similar messages [ 2097.184258] Lustre: Skipped 4 previous similar messages [ 2101.306574] Lustre: server umount lustre-MDT0000 complete [ 2102.247635] LustreError: 71143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2102.280427] LustreError: 71143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2104.899112] LustreError: 66543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788307914 with bad export cookie 13564733407602850466 [ 2104.905600] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2104.920899] LustreError: 66543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2105.337856] Lustre: server umount lustre-MDT0001 complete [ 2111.692504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2115.285588] Lustre: DEBUG MARKER: files compared the same [ 2127.573661] Lustre: server umount lustre-OST0000 complete [ 2143.612521] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 2146.494376] Key type lgssc unregistered [ 2146.873355] LNet: 74421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.886481] LNetError: 74421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.902641] LNet: Removed LNI 192.168.204.136@tcp [ 2147.849063] Key type .llcrypt unregistered [ 2147.856779] Key type ._llcrypt unregistered [ 2171.889081] Key type ._llcrypt registered [ 2171.890904] Key type .llcrypt registered [ 2171.980357] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2172.985141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2173.004597] alg: No test for adler32 (adler32-zlib) [ 2174.000111] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 2174.253564] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 2175.952499] Key type lgssc registered [ 2177.600405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2186.962245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2187.018827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.460129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2195.001304] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug +malloc all [ 2199.010881] LustreError: 75842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.022724] LustreError: 75842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2202.672449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.960702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2207.330717] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug +malloc all [ 2215.055610] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2221.193564] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2227.723577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2228.030094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.135878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2235.480226] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug +malloc all [ 2244.187478] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2257.892073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.894158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2257.911528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2262.132445] Lustre: server umount lustre-OST0000 complete [ 2268.128936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.148919] Lustre: Skipped 1 previous similar message [ 2268.155358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2268.160662] Lustre: Skipped 2 previous similar messages [ 2272.761392] Lustre: server umount lustre-MDT0000 complete [ 2273.248948] LustreError: 75842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.283262] LustreError: 75842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2277.051489] LustreError: 75822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788308086 with bad export cookie 17152261346040879645 [ 2277.055582] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.067056] LustreError: 75822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2277.511665] Lustre: server umount lustre-MDT0001 complete [ 2289.494178] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 2291.628837] Key type lgssc unregistered [ 2291.874919] LNet: 79381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2291.881137] LNetError: 79381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2291.893842] LNet: Removed LNI 192.168.204.136@tcp [ 2292.434459] Key type .llcrypt unregistered [ 2292.437366] Key type ._llcrypt unregistered [ 2320.409831] Key type ._llcrypt registered [ 2320.414200] Key type .llcrypt registered [ 2320.566376] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2322.215878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2322.246452] alg: No test for adler32 (adler32-zlib) [ 2323.283789] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 2323.566535] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 2325.344176] Key type lgssc registered [ 2326.744808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2334.469848] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2334.499299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2350.048946] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2355.169398] LustreError: 80773:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.136@tcp: failed processing log, type 4: rc = -110 [ 2381.857994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.545506] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2398.647682] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2407.193072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.287278] Lustre: *** cfs_fail_loc=706, val=0*** [ 2407.686901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.771625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2412.616687] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2413.029149] LustreError: 80798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.056435] LustreError: 80798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2417.134665] LustreError: 81859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.166493] LustreError: 81859:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2422.245863] LustreError: 80799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.273335] LustreError: 80799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2423.195178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.625170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2427.831445] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2437.177743] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2444.521162] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2448.866580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2448.886111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.918209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2449.380612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2453.363937] Lustre: server umount lustre-OST0000 complete [ 2459.617687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2459.625615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2459.634723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2459.634741] Lustre: Skipped 2 previous similar messages [ 2459.674369] Lustre: Skipped 2 previous similar messages [ 2464.369047] Lustre: server umount lustre-MDT0000 complete [ 2464.745128] LustreError: 81863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.768071] LustreError: 81863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2468.285025] LustreError: 80780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788308278 with bad export cookie 12715652468730158312 [ 2468.303420] LustreError: 80780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2468.306691] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2468.664578] Lustre: server umount lustre-MDT0001 complete [ 2481.162563] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 2483.738475] Key type lgssc unregistered [ 2484.062832] LNet: 84394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.071432] LNetError: 84394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2485.111289] LNet: Removed LNI 192.168.204.136@tcp [ 2485.929440] Key type .llcrypt unregistered [ 2485.934978] Key type ._llcrypt unregistered [ 2509.567035] Key type ._llcrypt registered [ 2509.569233] Key type .llcrypt registered [ 2509.690305] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2510.920494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2511.064124] alg: No test for adler32 (adler32-zlib) [ 2512.135422] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 2512.392825] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 2514.072198] Key type lgssc registered [ 2515.237341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2523.676268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2523.694304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.890484] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2536.972775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2537.679433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2543.082284] LustreError: 86498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.133669] LustreError: 86498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2543.791785] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2547.186388] LustreError: 86497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.211197] LustreError: 86497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2552.321111] LustreError: 86502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.342876] LustreError: 86502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2554.232322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2554.694770] LustreError: 86501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.738417] LustreError: 86501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2554.771354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2559.970318] LustreError: 86502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.989956] LustreError: 86502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2562.373864] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2568.162977] LustreError: 86501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.185091] LustreError: 86501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2572.944354] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2572.960657] Lustre: MGS: Client 6dc0c0bd-2baf-4ca5-8c65-01f46e028ad4 (at 0@lo) reconnecting [ 2573.020787] LustreError: 86512:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2573.354759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.453112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2579.467813] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2598.884082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2598.894680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2598.897252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2598.901413] Lustre: Skipped 1 previous similar message [ 2601.269781] Lustre: server umount lustre-OST0000 complete [ 2609.132458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.140152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.145236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2609.160467] Lustre: Skipped 1 previous similar message [ 2609.201072] Lustre: Skipped 2 previous similar messages [ 2611.379898] Lustre: server umount lustre-MDT0000 complete [ 2614.248234] LustreError: 86502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.265971] LustreError: 86502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2615.837826] Lustre: server umount lustre-MDT0001 complete [ 2620.149432] Lustre: server umount MGS complete [ 2632.701879] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:20:40 (1788308440) [ 2661.435963] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 2674.306784] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2715.492345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.262606] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2731.399057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2731.636757] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2731.685990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2731.765978] Lustre: lustre-MDT0001: new disk, initializing [ 2731.845096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2731.872429] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2731.882358] Lustre: 93691:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2731.889744] LustreError: 93691:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2735.870972] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2747.248217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2747.515939] Lustre: lustre-OST0000: new disk, initializing [ 2747.522892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2747.532092] Lustre: 94456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2747.612856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2752.874557] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2755.622155] Lustre: 94912:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ce287594 x1875177187092864/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788308581 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2763.078937] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2769.867358] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2769.876149] Lustre: MGS: Client 3e84033b-0565-496a-9e34-82ac07bc9d2c (at 0@lo) reconnecting [ 2769.904679] Lustre: 92959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2769.917195] LustreError: 93677:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2769.944973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2770.013945] Lustre: lustre-MDT0000: new disk, initializing [ 2770.160188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2770.208064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2770.233547] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2771.298240] Lustre: 95525:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b6cfaaee x1875177187099648/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788308597 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2774.762397] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2776.480990] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2780.461144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2780.476919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2780.557955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2806.245446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2806.252231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2806.269715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2808.808373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2808.808891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2811.845637] Lustre: server umount lustre-OST0000 complete [ 2819.041712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2819.056339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.069208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2819.085552] Lustre: Skipped 1 previous similar message [ 2819.118051] Lustre: Skipped 2 previous similar messages [ 2822.369165] Lustre: server umount lustre-MDT0000 complete [ 2824.180462] LustreError: 93663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.199952] LustreError: 93663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2833.909874] Lustre: server umount lustre-MDT0001 complete [ 2838.157676] Lustre: server umount MGS complete [ 2850.393184] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:24:18 (1788308658) [ 2867.283116] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 2870.003045] Key type lgssc unregistered [ 2870.326931] LNet: 98032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2870.333472] LNetError: 98032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2870.359599] LNet: Removed LNI 192.168.204.136@tcp [ 2871.268380] Key type .llcrypt unregistered [ 2871.271901] Key type ._llcrypt unregistered [ 2888.476611] Key type ._llcrypt registered [ 2888.483490] Key type .llcrypt registered [ 2888.603261] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 2889.620796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2889.754744] alg: No test for adler32 (adler32-zlib) [ 2890.885320] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 2891.153516] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 2892.808172] Key type lgssc registered [ 2894.045594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2902.413934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2902.451453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2903.819646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.390815] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2913.253920] LustreError: 99256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2913.281736] LustreError: 99256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2918.372824] LustreError: 99255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.601293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2919.194129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2924.488516] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2933.927797] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2939.862686] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2947.068797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2947.521216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2948.540242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2953.958251] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 2961.768675] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2974.178092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2974.181264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.201432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2974.224455] Lustre: Skipped 1 previous similar message [ 2976.934327] Lustre: server umount lustre-OST0000 complete [ 2984.421925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.424084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2984.448801] Lustre: Skipped 1 previous similar message [ 2984.456825] Lustre: Skipped 1 previous similar message [ 2989.537910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2989.546241] Lustre: Skipped 2 previous similar messages [ 2994.658795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2994.675140] Lustre: Skipped 1 previous similar message [ 2996.192186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2996.392372] Lustre: server umount lustre-MDT0000 complete [ 2999.781336] LustreError: 99255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2999.795475] LustreError: 99255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3000.814279] LustreError: 99236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788308810 with bad export cookie 8350200031658434757 [ 3000.831511] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3000.835795] LustreError: 99236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3001.235986] Lustre: server umount lustre-MDT0001 complete [ 3013.611300] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3016.163327] Key type lgssc unregistered [ 3016.431565] LNet: 102792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.436054] LNetError: 102792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3016.449182] LNet: Removed LNI 192.168.204.136@tcp [ 3017.187335] Key type .llcrypt unregistered [ 3017.190641] Key type ._llcrypt unregistered [ 3033.140374] Key type ._llcrypt registered [ 3033.144353] Key type .llcrypt registered [ 3033.275684] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3034.752863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3034.785746] alg: No test for adler32 (adler32-zlib) [ 3036.027482] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 3036.298518] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3037.992205] Key type lgssc registered [ 3039.300083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3046.002759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3046.017289] LustreError: 103874:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3046.049613] LustreError: 103878:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3046.055789] LustreError: 103874:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4972 [ 3046.061986] LustreError: 103874:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3046.073916] LustreError: 103874:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3046.079764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3047.834898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.609782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.900407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3061.646670] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3068.003205] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing lsmod [ 3071.368837] LustreError: 105085:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3071.369214] LustreError: 105086:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3071.379923] LustreError: 105085:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 3071.386854] LustreError: 105085:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3071.410626] LustreError: 105085:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3071.415540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3071.845227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.920535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3077.604779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3077.614147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.651965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3078.117865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.147722] Lustre: Skipped 1 previous similar message [ 3078.161718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3078.168244] Lustre: Skipped 2 previous similar messages [ 3081.252627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.266627] Lustre: Skipped 2 previous similar messages [ 3081.833968] Lustre: server umount lustre-MDT0000 complete [ 3086.064064] LustreError: 103895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788308895 with bad export cookie 564541723338727182 [ 3086.077980] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3086.080315] LustreError: 103895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3086.307912] LustreError: 103911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3086.317940] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.337195] LustreError: 103911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3086.344051] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3087.520606] LustreError: 104347:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000000e15b14 x1875177736525440/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3087.747310] Lustre: server umount lustre-MDT0001 complete [ 3091.816790] Lustre: server umount lustre-OST0000 complete [ 3101.091235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.576361] LustreError: 106148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3101.677314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3106.785502] LustreError: 106149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.979571] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3111.906831] LustreError: 106148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3117.031909] LustreError: 106149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.293403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.689758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3123.638204] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3132.263350] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3139.476668] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3147.716674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3148.347857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3155.704412] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3163.938455] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3171.888117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3182.049795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3182.058642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3182.069571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3182.088212] Lustre: Skipped 1 previous similar message [ 3186.341149] Lustre: server umount lustre-OST0000 complete [ 3190.246903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3190.252894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.267678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3190.276729] Lustre: Skipped 1 previous similar message [ 3196.616449] Lustre: server umount lustre-MDT0000 complete [ 3197.409368] LustreError: 106144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.431362] LustreError: 106144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3200.749245] LustreError: 106130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788309010 with bad export cookie 564541723338728113 [ 3200.770362] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3201.228360] Lustre: server umount lustre-MDT0001 complete [ 3213.864257] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3216.924792] Key type lgssc unregistered [ 3217.270814] LNet: 109709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3217.276736] LNetError: 109709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3217.289242] LNet: Removed LNI 192.168.204.136@tcp [ 3218.135977] Key type .llcrypt unregistered [ 3218.140365] Key type ._llcrypt unregistered [ 3245.394131] Key type ._llcrypt registered [ 3245.396132] Key type .llcrypt registered [ 3245.536556] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3246.568430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3246.611503] alg: No test for adler32 (adler32-zlib) [ 3247.624236] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 3247.909888] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3249.656187] Key type lgssc registered [ 3250.873444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3258.365812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3258.382805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.870214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.304553] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3270.125183] LustreError: 111129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.151684] LustreError: 111129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3273.755720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3274.254992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3279.533536] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3287.161924] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3292.468708] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3299.248506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3299.737906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3304.893048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3307.259551] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3316.776141] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3322.569721] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3326.645568] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3331.555176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3331.562762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.575349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3335.650933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3335.655396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3335.657076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3335.671434] Lustre: Skipped 1 previous similar message [ 3340.773020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3340.802770] Lustre: Skipped 1 previous similar message [ 3345.377576] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3345.577338] Lustre: server umount lustre-OST0000 complete [ 3351.009808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3351.013978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3351.025996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3351.048673] Lustre: Skipped 1 previous similar message [ 3356.129897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3356.141039] Lustre: Skipped 2 previous similar messages [ 3364.320150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3364.534228] Lustre: server umount lustre-MDT0000 complete [ 3366.371881] LustreError: 111130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.410581] LustreError: 111130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3368.774508] LustreError: 111111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788309178 with bad export cookie 662093483794775817 [ 3368.777308] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3368.789473] LustreError: 111111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3369.188850] Lustre: server umount lustre-MDT0001 complete [ 3379.803686] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3382.977853] Key type lgssc unregistered [ 3383.406623] LNet: 114773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3383.432287] LNetError: 114773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3383.459215] LNet: Removed LNI 192.168.204.136@tcp [ 3384.465776] Key type .llcrypt unregistered [ 3384.469322] Key type ._llcrypt unregistered [ 3403.162213] Key type ._llcrypt registered [ 3403.165317] Key type .llcrypt registered [ 3403.266763] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3404.435989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3404.461067] alg: No test for adler32 (adler32-zlib) [ 3405.639490] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 3405.856649] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3407.528586] Key type lgssc registered [ 3408.568743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3415.276197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3415.291986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.879566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.476666] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3426.274627] LustreError: 115998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.297770] LustreError: 115998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3431.396514] LustreError: 115997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3431.752759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.245888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3436.088654] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3443.726860] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3449.870522] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3457.561655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3458.135543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3461.260608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3465.402418] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3474.037135] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3487.226620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3487.228669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3487.249106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3487.260487] Lustre: Skipped 1 previous similar message [ 3489.560935] Lustre: server umount lustre-OST0000 complete [ 3493.862574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3493.872199] LustreError: Skipped 1 previous similar message [ 3493.875188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3493.894144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3493.911232] Lustre: Skipped 1 previous similar message [ 3497.441346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3497.444409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3497.460920] Lustre: Skipped 1 previous similar message [ 3499.774413] Lustre: server umount lustre-MDT0000 complete [ 3502.569752] LustreError: 117534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3502.595169] LustreError: 117534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3503.683268] LustreError: 115979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788309313 with bad export cookie 18295710494617611928 [ 3503.697808] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3503.702326] LustreError: 115979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3504.241665] Lustre: server umount lustre-MDT0001 complete [ 3515.037158] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3517.479339] Key type lgssc unregistered [ 3517.821828] LNet: 119536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3517.829538] LNetError: 119536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3517.853790] LNet: Removed LNI 192.168.204.136@tcp [ 3518.698873] Key type .llcrypt unregistered [ 3518.702201] Key type ._llcrypt unregistered [ 3541.582795] Key type ._llcrypt registered [ 3541.585769] Key type .llcrypt registered [ 3541.727784] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3542.770417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3542.780653] alg: No test for adler32 (adler32-zlib) [ 3543.826720] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 3544.142586] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3545.816202] Key type lgssc registered [ 3546.924616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3554.819390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3554.843996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.405864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3561.539789] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3566.561193] LustreError: 120955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.584373] LustreError: 120955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3569.977815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.392098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3574.844360] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3582.801679] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3589.100448] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3595.777025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3596.319163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3601.456582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3602.345446] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3610.008663] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3619.554977] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3623.092967] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3623.100718] Lustre: Skipped 2 previous similar messages [ 3626.633343] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3626.687122] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3627.929870] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3627.937092] Lustre: Skipped 2 previous similar messages [ 3634.216967] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3634.220570] Lustre: Skipped 1 previous similar message [ 3639.493331] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.36@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3639.501972] Lustre: Skipped 2 previous similar messages [ 3642.281168] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.36@tcp 192.168.0.[2,10]@tcp [ 3646.348673] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.36/24@tcp in log lustre-MDT0000 [ 3646.353848] Lustre: Skipped 2 previous similar messages [ 3651.648161] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.36/24@tcp [ 3651.659638] Lustre: Skipped 1 previous similar message [ 3658.727243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3658.728531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3658.754460] LustreError: Skipped 2 previous similar messages [ 3658.785098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3661.800678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3661.803564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3661.815445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3661.829262] LustreError: Skipped 2 previous similar messages [ 3664.807682] Lustre: server umount lustre-OST0000 complete [ 3667.939887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3667.944672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3667.967052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3667.969495] Lustre: Skipped 1 previous similar message [ 3672.033520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3672.034419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.154497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3677.160176] Lustre: Skipped 2 previous similar messages [ 3682.272284] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3682.551304] Lustre: server umount lustre-MDT0000 complete [ 3686.370442] LustreError: 123446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.399211] LustreError: 123446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3687.644144] LustreError: 120936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788309497 with bad export cookie 9310669128581485598 [ 3687.651102] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3687.659714] LustreError: 120936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3688.234661] Lustre: server umount lustre-MDT0001 complete [ 3700.761224] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3703.980759] Key type lgssc unregistered [ 3704.302557] LNet: 125854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3704.311227] LNetError: 125854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3704.326774] LNet: Removed LNI 192.168.204.136@tcp [ 3705.233314] Key type .llcrypt unregistered [ 3705.234899] Key type ._llcrypt unregistered [ 3734.640800] Key type ._llcrypt registered [ 3734.644334] Key type .llcrypt registered [ 3734.763513] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3735.962221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3736.088410] alg: No test for adler32 (adler32-zlib) [ 3737.510512] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 3737.834291] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3739.600225] Key type lgssc registered [ 3741.132577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3750.573978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3750.592171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.156857] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3752.166791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3752.205163] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.36/24@tcp [ 3757.387150] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3761.633727] LustreError: 127416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3761.669474] LustreError: 127416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3766.753662] LustreError: 127417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.068031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3768.509196] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3768.537112] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.36/24@tcp [ 3768.589022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3774.228803] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3782.445713] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3788.456241] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3795.017869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3795.348423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.408207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3802.206596] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3809.966568] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3824.098435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.105945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3824.119061] Lustre: Skipped 1 previous similar message [ 3824.139065] Lustre: Skipped 1 previous similar message [ 3829.220540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.228375] Lustre: Skipped 1 previous similar message [ 3829.657440] Lustre: server umount lustre-OST0000 complete [ 3834.341084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3834.374383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3834.384799] Lustre: Skipped 1 previous similar message [ 3839.458508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.464226] Lustre: Skipped 1 previous similar message [ 3840.358424] Lustre: server umount lustre-MDT0000 complete [ 3844.329770] LustreError: 127397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788309654 with bad export cookie 13830568227391992210 [ 3844.343943] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3844.672808] Lustre: server umount lustre-MDT0001 complete [ 3856.446790] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 3859.258199] Key type lgssc unregistered [ 3859.556702] LNet: 131011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3859.562393] LNetError: 131011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3859.586618] LNet: Removed LNI 192.168.204.136@tcp [ 3860.453367] Key type .llcrypt unregistered [ 3860.457856] Key type ._llcrypt unregistered [ 3891.032228] Key type ._llcrypt registered [ 3891.037481] Key type .llcrypt registered [ 3891.208282] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 3906.457879] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 3907.833617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3907.856929] alg: No test for adler32 (adler32-zlib) [ 3909.009851] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 3909.384560] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 3911.089455] Key type lgssc registered [ 3912.351680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3949.628984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3949.654949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3950.965616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3951.034439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3951.162928] Lustre: lustre-MDT0000: new disk, initializing [ 3951.263475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3951.290647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3956.824673] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3968.299951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3968.383813] Lustre: 134414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3968.463466] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3968.467909] Lustre: Skipped 1 previous similar message [ 3968.576992] Lustre: lustre-MDT0001: new disk, initializing [ 3968.702200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3968.763251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3968.773327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3973.239381] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 3982.732189] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3988.978966] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3995.314298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3995.557451] Lustre: lustre-OST0000: new disk, initializing [ 3995.567236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3995.578317] Lustre: 136382:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3995.669697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4000.670401] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4001.345971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4001.373091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4001.501795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4010.345164] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4015.992574] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4016.382829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4021.451216] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4021.716263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4027.365728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4027.367586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4027.369426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4027.383327] Lustre: Skipped 1 previous similar message [ 4027.409539] LustreError: Skipped 1 previous similar message [ 4032.490396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4032.507041] Lustre: Skipped 2 previous similar messages [ 4037.607049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4037.628719] Lustre: Skipped 1 previous similar message [ 4038.112187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4038.301384] Lustre: server umount lustre-OST0000 complete [ 4045.800927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4045.811094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.826887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4047.843932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.745880] Lustre: server umount lustre-MDT0000 complete [ 4054.526996] LustreError: 134409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788309864 with bad export cookie 16274303493819141021 [ 4054.536151] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4054.545975] LustreError: 134409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4054.962440] Lustre: server umount lustre-MDT0001 complete [ 4064.435654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4064.877527] LustreError: 138603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.894385] LustreError: 138603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4064.970961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4069.949413] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4070.373772] LustreError: 138603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.468166] LustreError: 138604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4079.586874] LustreError: 138603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4079.914547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4080.448209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4085.543447] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4093.946964] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4100.228968] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4107.814563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4108.167217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4114.878388] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4122.448526] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4126.693220] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4126.883927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4131.031696] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4131.200478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4140.324220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4140.505493] Lustre: lustre-OST0001: new disk, initializing [ 4140.514826] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4140.522658] Lustre: 141982:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4140.647211] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4146.238364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4146.250194] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4146.334658] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4146.777559] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4154.710120] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4154.941567] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4159.617229] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4159.883506] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4166.121722] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94fa12c34800.ost_server_uuid 50 [ 4167.631510] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff94fa12c34800.ost_server_uuid in IDLE state after 0 sec [ 4177.397358] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4177.403692] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.406962] LustreError: Skipped 1 previous similar message [ 4177.409152] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4177.419662] Lustre: Skipped 1 previous similar message [ 4177.424407] Lustre: Skipped 2 previous similar messages [ 4178.491747] Lustre: server umount lustre-OST0001 complete [ 4182.501244] LustreError: 140432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.503316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4182.543506] LustreError: 140432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4187.636506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4187.646094] Lustre: Skipped 4 previous similar messages [ 4188.808878] Lustre: server umount lustre-OST0000 complete [ 4197.867721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.889451] Lustre: Skipped 2 previous similar messages [ 4199.887054] Lustre: server umount lustre-MDT0000 complete [ 4202.993376] LustreError: 138603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4203.014736] LustreError: 138603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4204.480264] LustreError: 138585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310014 with bad export cookie 16274303493819141896 [ 4204.485781] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4204.492017] LustreError: 138585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4204.835801] Lustre: server umount lustre-MDT0001 complete [ 4226.547408] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4229.863694] Key type lgssc unregistered [ 4230.170674] LNet: 144992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4230.180950] LNetError: 144992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4231.206882] LNet: Removed LNI 192.168.204.136@tcp [ 4232.099908] Key type .llcrypt unregistered [ 4232.104247] Key type ._llcrypt unregistered [ 4275.409184] Key type ._llcrypt registered [ 4275.412036] Key type .llcrypt registered [ 4275.532169] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 4288.635299] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4289.543214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4289.600766] alg: No test for adler32 (adler32-zlib) [ 4290.593808] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 4290.822084] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4292.456222] Key type lgssc registered [ 4293.614329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4325.232565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4325.285635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4326.524176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4326.553270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4326.629561] Lustre: lustre-MDT0000: new disk, initializing [ 4326.750446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4326.773927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4331.159860] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4340.940664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.074685] Lustre: 149126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4341.120959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4341.124571] Lustre: Skipped 1 previous similar message [ 4341.178558] Lustre: lustre-MDT0001: new disk, initializing [ 4341.241914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4341.269460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4341.281152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4345.453397] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4353.844201] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4361.605214] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4369.235173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4369.644960] Lustre: lustre-OST0000: new disk, initializing [ 4369.658369] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4369.681375] Lustre: 151096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4369.833754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4376.440910] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4378.682346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4378.690770] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4378.804291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4385.546730] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4389.422149] Lustre: Failing over lustre-OST0000 [ 4389.539125] Lustre: server umount lustre-OST0000 complete [ 4406.953521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4407.255964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.299168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4407.318819] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4407.829748] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4408.289115] Lustre: 147352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788310201/real 1788310201] req@00000000b2876dd4 x1875179051450624/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788310217 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4410.336367] Lustre: 147354:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788310204/real 1788310204] req@000000000cc5daec x1875179051451904/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788310220 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4410.383363] Lustre: 147354:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4412.445404] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4412.445412] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4412.872952] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4415.456150] Lustre: 147351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788310209/real 1788310209] req@00000000789bca43 x1875179051453184/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788310225 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4415.484162] Lustre: 147351:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4416.322378] Lustre: Failing over lustre-MDT0000 [ 4416.593773] Lustre: server umount lustre-MDT0000 complete [ 4417.513049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4417.525894] Lustre: Skipped 3 previous similar messages [ 4420.640272] Lustre: 147352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788310214/real 1788310214] req@0000000005187c42 x1875179051454208/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788310230 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4422.638638] LustreError: 150739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4422.649198] LustreError: 150739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4423.183121] LustreError: 150495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.746924] LustreError: 149138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.775479] LustreError: 149138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4432.869601] LustreError: 149139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4432.882332] LustreError: 149139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4433.888986] Lustre: 147351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788310227/real 1788310227] req@0000000097e2d453 x1875179051462528/t0(0) o400->MGC192.168.204.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788310243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4433.923725] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4436.104356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4438.536039] LustreError: 149134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4438.562316] LustreError: 149134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4444.342441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4444.394238] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4448.417162] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4448.773038] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4449.780854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4449.786535] Lustre: Skipped 1 previous similar message [ 4449.809339] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4449.842112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4458.466336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4458.485159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4458.516978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4460.011345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4460.027387] Lustre: Skipped 1 previous similar message [ 4462.513455] Lustre: server umount lustre-OST0000 complete [ 4470.242184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4470.242548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4470.245989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4470.282382] Lustre: Skipped 2 previous similar messages [ 4472.687912] Lustre: server umount lustre-MDT0000 complete [ 4475.363052] LustreError: 149133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.394676] LustreError: 149133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4476.014924] LustreError: 149119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310285 with bad export cookie 6229933067260323423 [ 4476.016184] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4476.023387] LustreError: 149119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4476.301189] Lustre: server umount lustre-MDT0001 complete [ 4486.021478] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 4488.450808] Key type lgssc unregistered [ 4488.710644] LNet: 154754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4488.719026] LNetError: 154754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4488.745137] LNet: Removed LNI 192.168.204.136@tcp [ 4489.527630] Key type .llcrypt unregistered [ 4489.529853] Key type ._llcrypt unregistered [ 4523.038624] Key type ._llcrypt registered [ 4523.041700] Key type .llcrypt registered [ 4523.184502] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4524.092615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4524.176094] alg: No test for adler32 (adler32-zlib) [ 4525.284642] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 4525.615291] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 4527.312304] Key type lgssc registered [ 4528.465877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4535.682614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4535.697335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4537.203870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4541.877831] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4547.556969] LustreError: 156274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4547.582083] LustreError: 156274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4551.153623] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4552.674029] LustreError: 156275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4555.322128] LustreError: 156303:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4555.340329] LustreError: 156303:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4555.348719] LustreError: 156303:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4555.787559] Lustre: server umount lustre-MDT0000 complete [ 4570.180236] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 4580.220371] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 4614.027629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4614.204839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4614.227075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4614.288629] Lustre: lustre-MDT0000: new disk, initializing [ 4614.355436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4614.371469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4619.013824] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4628.496487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4628.640642] Lustre: 159604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4628.730434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4628.739136] Lustre: Skipped 1 previous similar message [ 4628.801254] Lustre: lustre-MDT0001: new disk, initializing [ 4628.893500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4628.938056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4628.959442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4633.529301] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4642.446587] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4648.192464] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4653.908532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4654.140824] Lustre: lustre-OST0000: new disk, initializing [ 4654.143901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4654.151524] Lustre: 161572:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4654.228307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4659.375641] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 4663.841907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4663.851469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4663.959354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4667.269868] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4670.444716] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4672.116984] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:54:39 (1788310479) [ 4935.649730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4935.651762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4935.658665] Lustre: Skipped 1 previous similar message [ 4938.057834] Lustre: server umount lustre-MDT0000 complete [ 4940.770212] LustreError: 159617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4940.798557] LustreError: 159617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4941.618225] LustreError: 159597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310751 with bad export cookie 3301471278176647934 [ 4941.628912] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4941.637246] LustreError: 159597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4945.891333] LustreError: 159613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4945.901316] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4945.925236] Lustre: Skipped 1 previous similar message [ 4945.928239] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4945.931774] Lustre: Skipped 2 previous similar messages [ 4948.020882] Lustre: server umount lustre-MDT0001 complete [ 4966.368178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4966.598202] Lustre: server umount lustre-OST0000 complete [ 4976.318678] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 4986.039275] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5020.748485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.937947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5020.978467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5021.080621] Lustre: lustre-MDT0000: new disk, initializing [ 5021.154567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5021.170659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5025.683926] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5036.433471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5036.556699] Lustre: 166237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5036.616286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5036.622630] Lustre: Skipped 1 previous similar message [ 5036.694346] Lustre: lustre-MDT0001: new disk, initializing [ 5036.796244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5036.831843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5036.849863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5041.620768] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5050.635953] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5056.219397] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5062.721917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5062.927227] Lustre: lustre-OST0000: new disk, initializing [ 5062.930894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5062.937223] Lustre: 168216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5063.034476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5069.645562] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5072.448992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5072.467173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5072.625080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5079.636795] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5084.754731] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5085.003449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5089.596555] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5089.810673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5093.348478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5093.354283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5093.356894] LustreError: Skipped 1 previous similar message [ 5093.360154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5093.360162] Lustre: Skipped 1 previous similar message [ 5093.381805] Lustre: Skipped 1 previous similar message [ 5097.992460] Lustre: server umount lustre-OST0000 complete [ 5103.588556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5103.591387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.602405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5103.613131] Lustre: Skipped 1 previous similar message [ 5103.617760] Lustre: Skipped 1 previous similar message [ 5108.706604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5108.713347] Lustre: Skipped 1 previous similar message [ 5115.872560] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5116.095247] Lustre: server umount lustre-MDT0000 complete [ 5118.949042] LustreError: 166251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.975795] LustreError: 166251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5120.636982] LustreError: 166230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310930 with bad export cookie 3301471278176806764 [ 5120.672032] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5121.114171] Lustre: server umount lustre-MDT0001 complete [ 5130.928202] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:02:18 (1788310938) [ 5148.462370] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 5158.284248] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5195.766600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5196.124252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5196.202693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5196.286621] Lustre: lustre-MDT0000: new disk, initializing [ 5196.391062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5196.415660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5202.065867] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5214.719592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.880613] Lustre: 172973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5215.219538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5220.575683] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5230.476156] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5236.390967] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5243.670790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5244.060535] Lustre: lustre-OST0000: new disk, initializing [ 5244.069963] Lustre: Skipped 1 previous similar message [ 5244.075362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5244.086960] Lustre: Skipped 2 previous similar messages [ 5244.093749] Lustre: 174949:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5251.288289] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5253.205219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5253.215629] Lustre: Skipped 1 previous similar message [ 5253.218191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5253.353390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5262.275502] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5274.082161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5274.082858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.093990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5274.101858] LustreError: Skipped 1 previous similar message [ 5274.136132] Lustre: Skipped 3 previous similar messages [ 5276.554929] Lustre: server umount lustre-OST0000 complete [ 5281.761738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5284.322617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5284.340931] Lustre: Skipped 2 previous similar messages [ 5287.760273] Lustre: server umount lustre-MDT0000 complete [ 5289.442994] LustreError: 172980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5289.458085] LustreError: 172980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5291.771527] LustreError: 172963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311101 with bad export cookie 3301471278176807646 [ 5291.780310] LustreError: 172963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5291.783602] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5292.233563] Lustre: server umount lustre-MDT0001 complete [ 5300.414994] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:05:08 (1788311108) [ 5316.467776] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 5326.095382] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5363.055754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.361241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5363.400083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5363.506041] Lustre: lustre-MDT0000: new disk, initializing [ 5363.633628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5363.643217] Lustre: Skipped 2 previous similar messages [ 5363.659969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5368.071561] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5378.402534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5378.551981] Lustre: 179273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5378.813506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5383.568717] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5393.972289] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5400.421736] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5408.000320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5408.280455] Lustre: 181250:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5413.940797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5414.046898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5415.044492] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5425.871970] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5434.341169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5434.351375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5434.368233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5434.375830] Lustre: Skipped 3 previous similar messages [ 5440.538948] Lustre: server umount lustre-OST0000 complete [ 5445.602247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5445.614873] LustreError: Skipped 1 previous similar message [ 5451.812165] Lustre: server umount lustre-MDT0000 complete [ 5455.331160] LustreError: 179286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5455.347251] LustreError: 179286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5455.926610] LustreError: 179267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311265 with bad export cookie 3301471278176808731 [ 5455.951868] LustreError: 179267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5455.953271] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5456.422400] Lustre: server umount lustre-MDT0001 complete [ 5467.049867] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5470.118407] Key type lgssc unregistered [ 5470.354942] LNet: 183108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5470.372607] LNetError: 183108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5470.401953] LNet: Removed LNI 192.168.204.136@tcp [ 5471.193551] Key type .llcrypt unregistered [ 5471.196362] Key type ._llcrypt unregistered [ 5497.040404] Key type ._llcrypt registered [ 5497.042245] Key type .llcrypt registered [ 5497.134634] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5498.186724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5498.227070] alg: No test for adler32 (adler32-zlib) [ 5499.288300] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 5499.490544] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 5501.209839] Key type lgssc registered [ 5502.272825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5510.324460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5510.345286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5512.007955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5516.011596] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5521.386468] LustreError: 184527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5521.412893] LustreError: 184527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5525.616665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5526.120569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5531.600265] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5539.142660] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5546.346555] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5553.212559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5553.644115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5556.745995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5560.388360] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5567.880489] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5591.525929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.531856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5591.545101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5591.566922] Lustre: Skipped 1 previous similar message [ 5596.642053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5596.651444] Lustre: Skipped 1 previous similar message [ 5596.889544] Lustre: server umount lustre-OST0000 complete [ 5601.766746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5601.788878] Lustre: Skipped 1 previous similar message [ 5601.801541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5601.810103] Lustre: Skipped 1 previous similar message [ 5606.894081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5606.903152] Lustre: Skipped 1 previous similar message [ 5607.709759] Lustre: server umount lustre-MDT0000 complete [ 5612.004317] LustreError: 184524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5612.034806] LustreError: 184524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5612.293832] LustreError: 184508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311422 with bad export cookie 12567337517435167690 [ 5612.303148] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5612.306628] LustreError: 184508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5612.903710] Lustre: server umount lustre-MDT0001 complete [ 5626.119497] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 5629.616784] Key type lgssc unregistered [ 5629.919776] LNet: 188087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5629.926931] LNetError: 188087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5629.952238] LNet: Removed LNI 192.168.204.136@tcp [ 5630.986397] Key type .llcrypt unregistered [ 5630.991248] Key type ._llcrypt unregistered [ 5658.723681] Key type ._llcrypt registered [ 5658.726447] Key type .llcrypt registered [ 5658.873962] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 5660.608381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5660.666162] alg: No test for adler32 (adler32-zlib) [ 5661.664381] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 5661.881134] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 5663.600383] Key type lgssc registered [ 5664.752523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5672.089706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5672.112777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.408837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5676.741519] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5683.698433] LustreError: 189507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5683.718331] LustreError: 189507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5685.508101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.766818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5690.517405] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5701.890722] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5708.264477] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5715.170788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5715.434356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5718.527801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5720.809829] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5728.079763] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5738.980546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5738.994993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5739.009917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5740.060816] Lustre: lustre-OST0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 5741.032133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5741.034146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5745.165065] Lustre: lustre-OST0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 5745.169823] Lustre: Skipped 2 previous similar messages [ 5750.274353] Lustre: lustre-OST0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 5750.293216] Lustre: Skipped 2 previous similar messages [ 5751.265328] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5751.603689] Lustre: server umount lustre-OST0000 complete [ 5759.662495] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff94fa07b30800.ost_server_uuid 50 [ 5761.265587] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5776.874359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5776.878840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5776.885773] Lustre: Skipped 1 previous similar message [ 5776.897309] Lustre: Skipped 3 previous similar messages [ 5781.514477] Lustre: server umount lustre-MDT0000 complete [ 5781.985285] LustreError: 189502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5782.005374] LustreError: 189502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5783.012503] LustreError: 190505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5783.027430] LustreError: 190505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5785.388476] LustreError: 189487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311595 with bad export cookie 7004158509258263288 [ 5785.400142] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5785.404797] LustreError: 189487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5785.814692] Lustre: server umount lustre-MDT0001 complete [ 5794.529786] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:13:22 (1788311602) [ 5801.026691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.402799] LustreError: 193422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5801.514868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5806.351305] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5806.562344] LustreError: 193423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5816.367614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.884692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5821.730872] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5829.794191] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5835.850288] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5842.456923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5842.768136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5846.959456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5849.959686] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5858.335347] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5866.065608] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5866.326552] Lustre: lustre-OST0001: new disk, initializing [ 5866.330317] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5866.337332] Lustre: 196265:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5866.434832] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5872.371175] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5875.323171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5875.336655] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5875.407292] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5881.671215] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5885.948594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5885.981351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5885.992057] Lustre: Skipped 1 previous similar message [ 5890.613497] Lustre: server umount lustre-OST0000 complete [ 5891.045237] LustreError: 195247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5891.072183] LustreError: 195247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5896.357219] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5896.558775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5901.385401] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5901.591170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5907.987143] LustreError: 195245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5908.006659] LustreError: 195245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5916.642755] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5916.669345] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.685789] Lustre: Skipped 1 previous similar message [ 5916.689196] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5916.700414] Lustre: Skipped 2 previous similar messages [ 5921.828307] Lustre: server umount lustre-OST0001 complete [ 5926.885985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5926.903859] Lustre: Skipped 2 previous similar messages [ 5931.548874] Lustre: server umount lustre-MDT0000 complete [ 5934.467365] LustreError: 195268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311744 with bad export cookie 7004158509258264359 [ 5934.469928] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5934.481926] LustreError: 195268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5934.811317] Lustre: server umount lustre-MDT0001 complete [ 5954.737350] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:16:02 (1788311762) [ 5960.593507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5960.854983] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5960.936573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5961.234639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5965.523551] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5974.307715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5974.402826] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5974.444929] Lustre: 199703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5978.557694] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 5985.715949] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5992.454257] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6001.003391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6001.170373] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6008.746740] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6010.073768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6018.029699] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6025.528075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6025.690563] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6025.957643] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6025.962150] Lustre: Skipped 2 previous similar messages [ 6033.213476] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6041.083485] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6045.162056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.176071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6045.192416] Lustre: Skipped 1 previous similar message [ 6045.194847] Lustre: Skipped 2 previous similar messages [ 6050.280980] LustreError: 201990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6050.289128] LustreError: 201990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6050.324935] Lustre: server umount lustre-OST0000 complete [ 6069.201159] Lustre: server umount lustre-OST0001 complete [ 6084.175233] Lustre: server umount lustre-MDT0000 complete [ 6086.416186] Lustre: 188818:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788311880/real 1788311880] req@00000000596857e8 x1875180489176064/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788311896 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6086.440252] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6086.448242] Lustre: Skipped 2 previous similar messages [ 6088.020062] LustreError: 200461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311897 with bad export cookie 7004158509258265759 [ 6088.026868] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6088.040055] LustreError: 200461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6088.451153] Lustre: server umount lustre-MDT0001 complete [ 6111.931321] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:18:39 (1788311919) [ 6130.199855] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 6139.506459] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 6179.838452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.078549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6180.120979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6180.210227] Lustre: lustre-MDT0000: new disk, initializing [ 6180.304255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6180.327484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6184.511676] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6195.304743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6195.393392] Lustre: 207799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6195.432822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6195.437348] Lustre: Skipped 1 previous similar message [ 6195.515803] Lustre: lustre-MDT0001: new disk, initializing [ 6195.660227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6195.678446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6199.886908] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6210.782934] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6216.715641] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6224.472494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6224.785464] Lustre: lustre-OST0000: new disk, initializing [ 6224.791873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6224.799504] Lustre: 209770:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6230.634442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6230.650771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6230.779971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6231.861074] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6242.615228] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6246.963788] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6247.189835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6251.468236] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6251.713398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6256.632669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6256.637339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6256.645457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6256.645466] Lustre: Skipped 6 previous similar messages [ 6256.649023] Lustre: Skipped 1 previous similar message [ 6256.674591] LustreError: Skipped 2 previous similar messages [ 6259.799413] Lustre: server umount lustre-OST0000 complete [ 6270.043930] Lustre: server umount lustre-MDT0000 complete [ 6271.970467] LustreError: 207811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6271.995430] LustreError: 207811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6274.150791] LustreError: 207792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312084 with bad export cookie 7004158509258267320 [ 6274.154879] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6274.169187] LustreError: 207792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6274.537213] Lustre: server umount lustre-MDT0001 complete [ 6282.424340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6287.337665] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6295.397549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6299.844090] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6307.119485] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6312.686950] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6318.970379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6324.447201] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6332.829648] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6337.706414] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6337.887632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6343.370417] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6343.583643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6346.209416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6351.641257] Lustre: server umount lustre-OST0000 complete [ 6358.790625] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6364.201606] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6393.548962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6399.081352] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.092488] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6400.224139] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6406.217469] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6412.067978] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6413.412151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.435286] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6418.922444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6418.929628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6418.945648] Lustre: Skipped 4 previous similar messages [ 6418.951566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6418.958671] Lustre: Skipped 6 previous similar messages [ 6422.639937] Lustre: server umount lustre-OST0000 complete [ 6433.118890] Lustre: server umount lustre-MDT0000 complete [ 6434.273788] LustreError: 211987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6434.302613] LustreError: 211987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6436.836954] LustreError: 211970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312246 with bad export cookie 7004158509258268209 [ 6436.850459] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6437.290771] Lustre: server umount lustre-MDT0001 complete [ 6446.853211] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:24:14 (1788312254) [ 6453.169912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6453.910700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6453.924380] Lustre: Skipped 6 previous similar messages [ 6458.348874] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6467.180271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6472.327892] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6479.659644] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6485.767352] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6492.900818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6500.259210] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6508.292185] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6513.563660] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6513.723191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6517.753238] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6518.022389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6525.431974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6525.586661] Lustre: lustre-OST0001: new disk, initializing [ 6525.592145] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6525.597516] Lustre: 221329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6533.400458] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6534.245499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6534.254535] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6534.329338] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6545.132521] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6551.095712] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6551.327984] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6555.704335] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6556.021911] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6560.233919] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6560.237949] LustreError: Skipped 2 previous similar messages [ 6564.292844] Lustre: server umount lustre-OST0001 complete [ 6570.082497] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6570.279078] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6574.826804] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6575.107192] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6605.125558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6610.926344] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6610.943041] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6610.948756] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6610.981063] Lustre: Skipped 1 previous similar message [ 6613.678208] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6622.746489] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6624.514345] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6631.394029] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6641.120417] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6641.304782] Lustre: server umount lustre-OST0001 complete [ 6666.618972] Lustre: server umount lustre-OST0000 complete [ 6675.426985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6675.437794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6675.449191] Lustre: Skipped 10 previous similar messages [ 6675.473692] Lustre: Skipped 15 previous similar messages [ 6676.889637] Lustre: server umount lustre-MDT0000 complete [ 6681.028819] LustreError: 217934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312490 with bad export cookie 7004158509258269175 [ 6681.039458] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6681.046855] LustreError: 217934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6695.392122] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6695.735799] Lustre: server umount lustre-MDT0001 complete [ 6722.051972] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:28:49 (1788312529) [ 6729.240131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.581628] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6729.629515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6734.870590] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6745.169940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6745.296607] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6745.366497] Lustre: 226574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6750.375709] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6758.603038] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6765.481552] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6773.694437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6774.120082] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6781.873820] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6790.538084] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6802.099665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6802.146517] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6810.435085] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6818.544709] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6820.080976] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6824.650823] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6824.895925] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6830.677686] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6830.991668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6837.025867] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94fa054a6000.ost_server_uuid 50 [ 6839.016101] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff94fa054a6000.ost_server_uuid in IDLE state after 0 sec [ 6840.113239] Lustre: Permanently deactivating lustre-OST0001 [ 6840.131506] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6844.077490] Lustre: Permanently reactivating lustre-OST0001 [ 6850.295361] Lustre: server umount lustre-OST0001 complete [ 6851.553109] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6851.562641] LustreError: Skipped 4 previous similar messages [ 6851.568592] LustreError: 228424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6851.605968] LustreError: 228424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 6860.287377] Lustre: server umount lustre-OST0000 complete [ 6870.653915] Lustre: server umount lustre-MDT0000 complete [ 6874.875196] LustreError: 229451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312684 with bad export cookie 7004158509258270463 [ 6874.880966] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6874.899040] LustreError: 229451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6875.400148] Lustre: server umount lustre-MDT0001 complete [ 6901.110941] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:31:48 (1788312708) [ 6908.975818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.320749] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6909.346466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6909.358819] Lustre: Skipped 5 previous similar messages [ 6913.958767] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6923.564385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.669129] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6923.694631] Lustre: 233275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6928.476751] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6937.153488] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6943.763815] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6954.097731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6954.334256] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6954.359481] Lustre: Permanently deactivating lustre-OST0000 [ 6954.381482] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6962.196254] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6962.314852] LustreError: 235446:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 6963.552961] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 6971.863727] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6981.514930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6981.612129] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6981.793509] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6981.797833] Lustre: Skipped 11 previous similar messages [ 6987.486370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6990.957905] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7000.304995] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7002.407829] Lustre: Permanently reactivating lustre-OST0000 [ 7002.427966] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7002.437791] Lustre: Skipped 2 previous similar messages [ 7010.394471] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7010.395239] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 48 seconds. I think it's dead, and I am evicting it. exp 000000009a218f0a, cur 1788312820 deadline 1788312802 last 1788312772 [ 7010.407332] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7010.410218] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7010.487969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7019.500911] Lustre: server umount lustre-OST0001 complete [ 7023.074511] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7023.088506] LustreError: Skipped 2 previous similar messages [ 7030.107973] Lustre: server umount lustre-OST0000 complete [ 7040.687593] Lustre: server umount lustre-MDT0000 complete [ 7045.070604] LustreError: 233267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312854 with bad export cookie 7004158509258272143 [ 7045.076212] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7045.080732] LustreError: 233267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7045.366421] Lustre: server umount lustre-MDT0001 complete [ 7057.133325] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 7060.316456] Key type lgssc unregistered [ 7060.647640] LNet: 238161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7060.658870] LNetError: 238161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7060.674770] LNet: Removed LNI 192.168.204.136@tcp [ 7061.560854] Key type .llcrypt unregistered [ 7061.563265] Key type ._llcrypt unregistered [ 7085.536684] Key type ._llcrypt registered [ 7085.538889] Key type .llcrypt registered [ 7085.667648] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7087.172884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7087.232816] alg: No test for adler32 (adler32-zlib) [ 7088.494383] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 7088.830120] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 7090.531833] Key type lgssc registered [ 7091.814945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7102.794571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7102.811287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7104.495960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7110.827502] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7114.725599] LustreError: 239630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7114.749525] LustreError: 239630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7119.845416] LustreError: 239631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7121.812121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.969861] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7122.043618] Lustre: 239619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7122.055289] Lustre: Permanently deactivating lustre-MDT0001 [ 7122.069520] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7122.516710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7127.937422] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7129.643380] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7136.433199] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7143.421403] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7151.027403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7151.431194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7158.198681] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7166.955178] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7167.793804] LustreError: 241474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7167.805729] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7173.105770] LustreError: 241474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7175.164821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7175.334395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7175.564558] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7180.807885] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7180.808525] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7183.540910] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7191.924975] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7195.974264] Lustre: Permanently reactivating lustre-MDT0001 [ 7195.991970] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7196.001944] Lustre: Skipped 2 previous similar messages [ 7199.085407] Lustre: lustre-MDT0001: Client 7194d3a7-5346-46b6-bac2-6ab09b3d1369 (at 192.168.204.36@tcp) reconnecting [ 7202.391495] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7202.412037] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7202.418285] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7202.431041] LustreError: 243322:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7202.436512] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7202.440391] LustreError: 243322:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7202.440511] LustreError: 243322:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.136@tcp: cfg command failed: rc = -108 [ 7202.440531] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7202.440531] [ 7202.440818] LustreError: 239623:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7204.750100] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7206.453326] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7211.472812] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7211.652501] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7217.035910] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7217.336782] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7220.228121] Lustre: Permanently deactivating lustre-MDT0001 [ 7220.256188] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7220.265436] Lustre: Skipped 2 previous similar messages [ 7227.003665] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7234.955180] Lustre: server umount lustre-MDT0000 complete [ 7235.046524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7235.054972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7235.065483] LustreError: 239630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7235.080672] LustreError: 239630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7237.094828] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7237.107692] Lustre: Skipped 1 previous similar message [ 7238.918154] LustreError: 239611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313048 with bad export cookie 9213663743155907002 [ 7238.929479] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7238.930714] LustreError: 239611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7239.557322] Lustre: server umount lustre-MDT0001 complete [ 7245.434099] Lustre: server umount lustre-OST0000 complete [ 7258.462876] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:37:45 (1788313065) [ 7270.538319] Lustre: server umount lustre-OST0001 complete [ 7278.311931] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 7289.196505] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7327.037406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7327.319545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7327.328316] Lustre: Skipped 2 previous similar messages [ 7327.361737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7327.513570] Lustre: lustre-MDT0000: new disk, initializing [ 7327.639306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7327.667600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7332.137981] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7345.194952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.321108] Lustre: 247891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7345.389350] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7345.394699] Lustre: Skipped 1 previous similar message [ 7345.477956] Lustre: lustre-MDT0001: new disk, initializing [ 7345.600522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7345.668827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7345.685937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7350.739478] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7362.637577] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7368.781591] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7376.254676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7376.554183] Lustre: lustre-OST0000: new disk, initializing [ 7376.556915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7376.568399] Lustre: 249862:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7376.688853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7382.641221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7382.654211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7382.780875] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7383.367996] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7392.910805] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7395.342424] LustreError: 247899:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7405.432279] LustreError: 247899:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7405.455520] LustreError: 249504:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7405.600473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7405.819440] Lustre: lustre-OST0001: new disk, initializing [ 7405.823171] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7405.835431] Lustre: 250978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7411.873596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7411.885036] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7411.990753] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7413.615422] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7415.514893] LustreError: 249504:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7425.074033] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7426.924571] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7429.164616] Lustre: lustre-OST0001: Not available for connect from 192.168.204.36@tcp (stopping) [ 7432.673302] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7432.690856] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.722020] Lustre: Skipped 2 previous similar messages [ 7432.728246] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7432.750253] Lustre: Skipped 1 previous similar message [ 7434.250246] Lustre: lustre-OST0001: Not available for connect from 192.168.204.36@tcp (stopping) [ 7435.524339] Lustre: server umount lustre-OST0001 complete [ 7437.799511] LustreError: 249848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7437.820077] LustreError: 249848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7453.160975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7453.163597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7453.166199] Lustre: Skipped 1 previous similar message [ 7453.173168] Lustre: Skipped 1 previous similar message [ 7458.276397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7458.283564] LustreError: 249848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7458.289689] Lustre: Skipped 1 previous similar message [ 7458.300526] LustreError: 249848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7458.884842] Lustre: server umount lustre-OST0000 complete [ 7463.399165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.405206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7468.513748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7468.517912] Lustre: Skipped 3 previous similar messages [ 7477.216631] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7477.410472] Lustre: server umount lustre-MDT0000 complete [ 7481.273963] LustreError: 247882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313291 with bad export cookie 9213663743155909816 [ 7481.306060] LustreError: 247882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7481.320370] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7487.703965] Lustre: server umount lustre-MDT0001 complete [ 7499.894721] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 7502.854461] Key type lgssc unregistered [ 7503.450376] LNet: 253015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7503.466757] LNetError: 253015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7503.499646] LNet: Removed LNI 192.168.204.136@tcp [ 7504.198339] Key type .llcrypt unregistered [ 7504.200173] Key type ._llcrypt unregistered [ 7546.603265] Key type ._llcrypt registered [ 7546.605718] Key type .llcrypt registered [ 7546.702925] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7547.866247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7547.892118] alg: No test for adler32 (adler32-zlib) [ 7548.906610] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 7549.162518] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 7550.840295] Key type lgssc registered [ 7552.293128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7561.082893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7561.112755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7562.544237] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7562.577601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7562.773468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.080293] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7575.414521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7575.637146] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7575.702814] Lustre: 255225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7576.168044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7580.375493] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7588.666184] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7596.834305] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7604.967644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7605.201140] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7605.468963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7612.483957] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7614.086914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7621.650447] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7675.368532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7675.377154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7675.397702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7675.878247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7675.883337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7675.916605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7677.840349] Lustre: server umount lustre-OST0000 complete [ 7682.331652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7699.133704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7699.835716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7707.140941] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7715.189215] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7715.205664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7715.206309] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7715.220801] Lustre: Skipped 1 previous similar message [ 7715.789724] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7756.259870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7756.276966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7756.287906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7756.300418] Lustre: Skipped 1 previous similar message [ 7761.507324] Lustre: server umount lustre-OST0000 complete [ 7766.497759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7766.506207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7766.522507] Lustre: Skipped 2 previous similar messages [ 7771.618429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7771.624402] Lustre: Skipped 1 previous similar message [ 7780.320762] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7780.476286] Lustre: server umount lustre-MDT0000 complete [ 7781.858923] LustreError: 255232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7781.894620] LustreError: 255232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7784.657299] LustreError: 255217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313594 with bad export cookie 13934234985352393883 [ 7784.680313] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7785.050390] Lustre: server umount lustre-MDT0001 complete [ 7796.434291] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 7799.393489] Key type lgssc unregistered [ 7799.636959] LNet: 260654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7799.645856] LNetError: 260654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7799.664280] LNet: Removed LNI 192.168.204.136@tcp [ 7800.795140] Key type .llcrypt unregistered [ 7800.800079] Key type ._llcrypt unregistered [ 7825.813610] Key type ._llcrypt registered [ 7825.817196] Key type .llcrypt registered [ 7825.953293] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7827.214578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7827.264623] alg: No test for adler32 (adler32-zlib) [ 7828.353246] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 7828.603224] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 7830.280213] Key type lgssc registered [ 7831.503512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7840.207220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7840.219334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7841.768791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7845.829251] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7852.004528] LustreError: 262076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7852.032800] LustreError: 262076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7854.653758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7855.161866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7860.087375] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7868.491343] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7874.453484] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7882.501074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7883.078720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7887.185204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7890.117817] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 7898.678787] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7928.293261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7928.302968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7928.327658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7930.346930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7930.350048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7930.369496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7930.394062] Lustre: Skipped 1 previous similar message [ 7933.024446] Lustre: server umount lustre-OST0000 complete [ 7940.584020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7940.613820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7940.619448] Lustre: Skipped 1 previous similar message [ 7943.572411] Lustre: server umount lustre-MDT0000 complete [ 7945.697982] LustreError: 262072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7945.740740] LustreError: 262072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7947.524527] LustreError: 265631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313757 with bad export cookie 14986916177025974408 [ 7947.526906] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7947.537333] LustreError: 265631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7947.959433] Lustre: server umount lustre-MDT0001 complete [ 7960.403089] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 7963.449697] Key type lgssc unregistered [ 7963.940844] LNet: 266386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7963.951545] LNetError: 266386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7964.976403] LNet: Removed LNI 192.168.204.136@tcp [ 7966.154329] Key type .llcrypt unregistered [ 7966.156511] Key type ._llcrypt unregistered [ 7985.081762] Key type ._llcrypt registered [ 7985.095876] Key type .llcrypt registered [ 7985.278453] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 7986.836924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7986.892650] alg: No test for adler32 (adler32-zlib) [ 7988.279407] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 7988.652355] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 7990.392234] Key type lgssc registered [ 7991.859585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8002.062822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8002.079957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8003.739833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8009.070813] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8013.286416] LustreError: 267608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8013.300326] LustreError: 267608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8018.406240] LustreError: 267609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8018.613829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8019.216914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8024.545864] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8034.608814] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8042.254739] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8050.644983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8051.160351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8059.530288] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8068.554551] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8074.765485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8089.065116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8089.073997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8089.088510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8090.082725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8090.087125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8090.099166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8090.124938] Lustre: Skipped 1 previous similar message [ 8092.294826] Lustre: server umount lustre-OST0000 complete [ 8100.332938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.337415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8100.366643] Lustre: Skipped 1 previous similar message [ 8100.381598] Lustre: Skipped 1 previous similar message [ 8102.780431] Lustre: server umount lustre-MDT0000 complete [ 8105.446700] LustreError: 267608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8105.471649] LustreError: 267608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8106.650160] LustreError: 268342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313916 with bad export cookie 16062567096318174895 [ 8106.652135] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8106.662329] LustreError: 268342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8107.003423] Lustre: server umount lustre-MDT0001 complete [ 8118.152168] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 8120.931957] Key type lgssc unregistered [ 8121.456762] LNet: 271292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8121.478792] LNetError: 271292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8121.501840] LNet: Removed LNI 192.168.204.136@tcp [ 8122.570929] Key type .llcrypt unregistered [ 8122.575861] Key type ._llcrypt unregistered [ 8148.919417] Key type ._llcrypt registered [ 8148.920969] Key type .llcrypt registered [ 8149.014239] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 8150.143405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8150.175302] alg: No test for adler32 (adler32-zlib) [ 8151.222394] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 8151.471029] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 8153.145310] Key type lgssc registered [ 8154.240650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8161.091602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8161.104809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8162.552763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8166.680640] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8173.055812] LustreError: 272716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8173.072205] LustreError: 272716:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8174.654582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8174.992781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8179.233235] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8187.139231] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8193.725249] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8202.368274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8202.872332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8208.037420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8209.201883] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8216.966883] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8254.444747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8254.456693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.472704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8255.470512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8255.486513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8255.487688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8255.532774] Lustre: Skipped 1 previous similar message [ 8259.022581] Lustre: server umount lustre-OST0000 complete [ 8265.698495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8265.707438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8265.725482] Lustre: Skipped 1 previous similar message [ 8270.820126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8270.836384] Lustre: Skipped 2 previous similar messages [ 8271.834297] Lustre: server umount lustre-MDT0000 complete [ 8275.939806] LustreError: 272712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8275.972403] LustreError: 272712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8276.420578] LustreError: 272696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314086 with bad export cookie 13131780240927449281 [ 8276.423325] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8276.446389] LustreError: 272696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8277.026737] Lustre: server umount lustre-MDT0001 complete [ 8290.329657] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 8293.933916] Key type lgssc unregistered [ 8294.277570] LNet: 277028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8294.289888] LNetError: 277028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8294.307868] LNet: Removed LNI 192.168.204.136@tcp [ 8295.575375] Key type .llcrypt unregistered [ 8295.577088] Key type ._llcrypt unregistered [ 8315.230210] Key type ._llcrypt registered [ 8315.233274] Key type .llcrypt registered [ 8315.377664] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 8316.666457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8316.687788] alg: No test for adler32 (adler32-zlib) [ 8318.104497] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 8318.519527] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 8320.264190] Key type lgssc registered [ 8321.985876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8330.710097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8330.738635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8332.364917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8336.968088] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8342.515319] LustreError: 278280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8342.567775] LustreError: 278280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8347.419808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.620265] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8347.850809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8353.994726] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8362.955297] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8369.682460] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8377.463771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8377.856411] LustreError: 280128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8378.018709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8383.467425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8384.278988] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8392.091322] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8403.239810] Lustre: server umount lustre-OST0000 complete [ 8408.034985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8408.038554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8408.049175] Lustre: Skipped 1 previous similar message [ 8408.065503] Lustre: Skipped 1 previous similar message [ 8413.157988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8413.166645] Lustre: Skipped 1 previous similar message [ 8417.333566] Lustre: server umount lustre-MDT0000 complete [ 8418.273567] LustreError: 278263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8418.291436] LustreError: 278263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8421.273929] LustreError: 279041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314231 with bad export cookie 6275210812539049732 [ 8421.291300] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8421.292844] LustreError: 279041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8431.900211] Lustre: server umount lustre-MDT0001 complete [ 8443.623907] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 8447.539870] Key type lgssc unregistered [ 8447.892811] LNet: 281987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8447.905622] LNetError: 281987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8447.933911] LNet: Removed LNI 192.168.204.136@tcp [ 8449.037890] Key type .llcrypt unregistered [ 8449.040432] Key type ._llcrypt unregistered [ 8461.359536] Key type ._llcrypt registered [ 8461.361702] Key type .llcrypt registered [ 8461.482275] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8476.736771] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 8492.978750] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 8494.521699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8494.803073] alg: No test for adler32 (adler32-zlib) [ 8495.852907] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 8496.106714] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 8498.112185] Key type lgssc registered [ 8499.551103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8538.791988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8538.839110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8540.155684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8540.225457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8540.350959] Lustre: lustre-MDT0000: new disk, initializing [ 8540.497162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.514729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8545.549970] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8556.082274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8556.168733] Lustre: 285538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8556.246875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8556.253568] Lustre: Skipped 1 previous similar message [ 8556.328925] Lustre: lustre-MDT0001: new disk, initializing [ 8556.407223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8556.443093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8556.458034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8559.935552] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8571.616143] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8579.628035] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8588.204657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8588.761486] Lustre: lustre-OST0000: new disk, initializing [ 8588.767443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8588.779323] Lustre: 287508:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8588.952312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8596.611565] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8598.645578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8598.653348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8598.847839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8606.852836] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8612.779030] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8613.195495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8620.890909] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8621.306188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8624.611181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8624.613764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.626691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8624.646959] Lustre: Skipped 1 previous similar message [ 8624.674758] Lustre: Skipped 1 previous similar message [ 8629.730512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8630.697497] Lustre: server umount lustre-OST0000 complete [ 8634.850474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.863276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8634.874365] Lustre: Skipped 1 previous similar message [ 8634.883144] Lustre: Skipped 1 previous similar message [ 8639.970485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.981921] Lustre: Skipped 2 previous similar messages [ 8641.080917] Lustre: server umount lustre-MDT0000 complete [ 8645.440251] LustreError: 285533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314455 with bad export cookie 18173899601460857176 [ 8645.451357] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8645.461118] LustreError: 285533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8645.953950] Lustre: server umount lustre-MDT0001 complete [ 8654.956802] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 22:01:02 (1788314462) [ 8662.686269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8663.299167] LustreError: 289934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8663.328952] LustreError: 289934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8663.445328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8668.646935] LustreError: 289935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8669.592179] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8673.761939] LustreError: 289934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8678.885146] LustreError: 289935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8679.633402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8679.949873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8684.971586] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8694.314542] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8701.258750] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8709.617552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8709.981144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8718.433570] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8728.504984] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8750.051420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8750.054930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8750.077284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8751.394783] Lustre: server umount lustre-OST0000 complete [ 8756.712175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8756.724378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8756.741950] Lustre: Skipped 1 previous similar message [ 8760.293345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8760.311722] Lustre: Skipped 3 previous similar messages [ 8762.763560] Lustre: server umount lustre-MDT0000 complete [ 8765.417064] LustreError: 290663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8765.446672] LustreError: 290663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8768.750089] LustreError: 289916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314578 with bad export cookie 18173899601460858051 [ 8768.758213] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8768.761075] LustreError: 289916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8769.439727] Lustre: server umount lustre-MDT0001 complete [ 8785.443181] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 8789.157405] Key type lgssc unregistered [ 8789.480897] LNet: 293498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8789.492514] LNetError: 293498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8789.514650] LNet: Removed LNI 192.168.204.136@tcp [ 8790.762325] Key type .llcrypt unregistered [ 8790.770568] Key type ._llcrypt unregistered [ 8821.578619] Key type ._llcrypt registered [ 8821.585916] Key type .llcrypt registered [ 8821.754397] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 8838.365757] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 8839.596856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8839.784421] alg: No test for adler32 (adler32-zlib) [ 8840.924323] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 8841.359672] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 8843.056189] Key type lgssc registered [ 8844.590368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8901.905835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8901.922837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8903.284192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8903.313621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8903.385443] Lustre: lustre-MDT0000: new disk, initializing [ 8903.448389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8903.469838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8909.624159] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8921.947728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8922.076441] Lustre: 297327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8922.161506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8922.165957] Lustre: Skipped 1 previous similar message [ 8922.307606] Lustre: lustre-MDT0001: new disk, initializing [ 8922.419714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8922.474693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8922.493768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8928.162544] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8941.248672] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8947.807975] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8956.171689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8956.576524] Lustre: lustre-OST2710: new disk, initializing [ 8956.584718] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8956.591803] Lustre: 299297:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8956.743038] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8965.441635] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8966.241804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8966.251877] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8966.386154] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8978.153867] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8987.079047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8987.286292] Lustre: lustre-OST03e8: new disk, initializing [ 8987.297371] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8987.315079] Lustre: 300363:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8987.458946] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8995.117120] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 8997.481137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8997.501062] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8997.687085] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9004.214089] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9011.751387] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9011.977945] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9018.131176] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9018.443438] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9024.651667] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9024.878829] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9030.224644] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9030.464677] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9038.820185] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9038.838536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9038.853556] Lustre: Skipped 2 previous similar messages [ 9038.881883] Lustre: Skipped 1 previous similar message [ 9043.378464] Lustre: server umount lustre-MDT0000 complete [ 9049.063051] LustreError: 297333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9049.098206] LustreError: 297333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 9049.221794] LustreError: 297319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314859 with bad export cookie 13225257466388225716 [ 9049.226375] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9049.235765] LustreError: 297319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9049.617711] Lustre: server umount lustre-MDT0001 complete [ 9065.223258] Lustre: server umount lustre-OST2710 complete [ 9080.211713] Lustre: server umount lustre-OST03e8 complete [ 9102.834476] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [ 9113.215651] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 9155.306814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9155.551110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9155.590101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9155.699187] Lustre: lustre-MDT0000: new disk, initializing [ 9155.803174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9155.824968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9160.842886] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9172.599931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9172.746358] Lustre: 305621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9172.798876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9172.801684] Lustre: Skipped 1 previous similar message [ 9172.900556] Lustre: lustre-MDT0001: new disk, initializing [ 9173.013690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9173.055269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9173.077672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9178.048392] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9188.772039] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9195.510887] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9204.051064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9204.289984] Lustre: lustre-OST0000: new disk, initializing [ 9204.294103] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9204.300526] Lustre: 307589:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9204.408254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9212.318452] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9213.043732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9213.071452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9213.234093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9223.249980] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9229.208844] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9229.439320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9234.634317] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9234.866534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9239.015509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9239.016432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9239.026426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9239.026432] Lustre: Skipped 2 previous similar messages [ 9239.026435] Lustre: Skipped 2 previous similar messages [ 9239.061385] LustreError: Skipped 1 previous similar message [ 9244.138704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9244.144622] Lustre: Skipped 2 previous similar messages [ 9249.267135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9249.283796] Lustre: Skipped 1 previous similar message [ 9251.296281] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9251.549837] Lustre: server umount lustre-OST0000 complete [ 9259.495296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9259.496221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9259.513668] Lustre: Skipped 1 previous similar message [ 9259.528994] Lustre: Skipped 1 previous similar message [ 9262.757464] Lustre: server umount lustre-MDT0000 complete [ 9264.619743] LustreError: 305631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9264.645443] LustreError: 305631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9267.555042] LustreError: 305612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315077 with bad export cookie 13225257466388227410 [ 9267.564311] LustreError: 305612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9267.565099] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9267.860847] Lustre: server umount lustre-MDT0001 complete [ 9278.966913] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:11:26 (1788315086) [ 9280.935832] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9283.231964] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:11:30 (1788315090) [ 9287.786699] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 9320.395481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9320.897727] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9320.931257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9321.206349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9326.673952] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9336.278431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9336.488671] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9336.513934] Lustre: 311448:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9342.023063] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9365.744600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9366.004617] Lustre: Denying initial registration attempt from nid 192.168.204.136@tcp, specified as failover [ 9366.018032] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9366.027339] LustreError: 312900:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9366.041264] LustreError: 312900:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9366.060177] LustreError: 312900:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9366.201888] Lustre: server umount lustre-OST0000 complete [ 9366.205781] LustreError: 312900:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9374.185408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9374.220794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9374.233962] Lustre: Skipped 1 previous similar message [ 9379.848379] Lustre: server umount lustre-MDT0000 complete [ 9384.428277] LustreError: 311454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9384.442091] LustreError: 311454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9384.624868] LustreError: 311438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315194 with bad export cookie 13225257466388228292 [ 9384.637864] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9385.065440] Lustre: server umount lustre-MDT0001 complete [ 9396.500734] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 9400.035293] Key type lgssc unregistered [ 9400.330908] LNet: 314024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9400.338614] LNetError: 314024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9400.356549] LNet: Removed LNI 192.168.204.136@tcp [ 9401.133971] Key type .llcrypt unregistered [ 9401.136578] Key type ._llcrypt unregistered [ 9414.103754] Key type ._llcrypt registered [ 9414.107361] Key type .llcrypt registered [ 9414.336752] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 9415.552785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9415.583236] alg: No test for adler32 (adler32-zlib) [ 9416.790987] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 9417.125652] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 9418.848304] Key type lgssc registered [ 9420.223130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9473.418663] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 9483.826746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9483.865549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9485.390266] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9485.433552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9485.689493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9491.481093] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9501.407130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.547488] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9501.590819] Lustre: 316789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9501.906134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9506.492752] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9513.951848] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9520.296724] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9541.363899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9541.595671] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9541.861046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9548.418603] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9558.536386] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9561.060235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9561.073424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9561.087642] Lustre: Skipped 1 previous similar message [ 9566.185225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9566.192511] Lustre: Skipped 1 previous similar message [ 9571.310662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9571.319803] Lustre: Skipped 1 previous similar message [ 9575.392137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9575.744266] Lustre: server umount lustre-OST0000 complete [ 9581.538554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9581.543458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9581.559173] Lustre: Skipped 2 previous similar messages [ 9581.578104] Lustre: Skipped 1 previous similar message [ 9586.719715] Lustre: server umount lustre-MDT0000 complete [ 9591.001227] LustreError: 316782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315400 with bad export cookie 7778626256642150712 [ 9591.007529] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9591.011374] LustreError: 316782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9591.576484] Lustre: server umount lustre-MDT0001 complete [ 9605.291953] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 9608.771328] Key type lgssc unregistered [ 9609.128310] LNet: 320373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9609.132643] LNetError: 320373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9609.146533] LNet: Removed LNI 192.168.204.136@tcp [ 9610.210094] Key type .llcrypt unregistered [ 9610.213341] Key type ._llcrypt unregistered [ 9639.880944] Key type ._llcrypt registered [ 9639.884289] Key type .llcrypt registered [ 9639.997806] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 9641.339680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9641.457390] alg: No test for adler32 (adler32-zlib) [ 9642.632228] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 9642.926925] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 9644.664223] Key type lgssc registered [ 9646.210644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9654.812554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9654.862091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9656.359125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9661.350432] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9666.530488] LustreError: 321790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9666.541575] LustreError: 321790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9670.644510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9670.939401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9676.555724] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9685.098116] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9690.882367] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9700.270290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9700.744866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9707.361941] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9714.077754] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9725.411843] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9725.415288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9725.432301] Lustre: Skipped 1 previous similar message [ 9725.451921] Lustre: Skipped 2 previous similar messages [ 9727.004204] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.36@tcp (stopping) [ 9729.594476] Lustre: server umount lustre-MDT0000 complete [ 9730.536711] LustreError: 321784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9730.577037] LustreError: 321784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9732.103702] LustreError: 321785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9733.368741] LustreError: 321771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315543 with bad export cookie 5848141355870532948 [ 9733.380171] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9733.383533] LustreError: 321771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9735.661615] LustreError: 321786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9735.662871] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9735.701746] Lustre: Skipped 1 previous similar message [ 9735.708572] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9738.664516] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9738.671548] Lustre: Skipped 1 previous similar message [ 9739.882534] Lustre: server umount lustre-MDT0001 complete [ 9746.520119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9755.938643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9758.751275] LustreError: 323622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9758.783506] LustreError: 323622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9758.921279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9759.038405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9763.804263] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9769.126773] LustreError: 325386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9769.152806] LustreError: 325386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9769.158326] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9774.708328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9775.087165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9780.015691] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9780.225892] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9789.020466] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9791.146507] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9798.054680] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9800.045612] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9809.889210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9809.901886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9809.912473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9810.929577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9812.346953] Lustre: server umount lustre-OST0000 complete [ 9821.153548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9821.157655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9821.163529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9821.179902] Lustre: Skipped 2 previous similar messages [ 9821.193193] Lustre: Skipped 2 previous similar messages [ 9822.725221] Lustre: server umount lustre-MDT0000 complete [ 9826.279103] LustreError: 325387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9826.306464] LustreError: 325387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9826.798604] LustreError: 323634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315636 with bad export cookie 5848141355870540515 [ 9826.805359] LustreError: 323634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9826.806180] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9827.176405] Lustre: server umount lustre-MDT0001 complete [ 9837.651827] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [ 9841.118059] Key type lgssc unregistered [ 9841.473458] LNet: 328067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9841.488438] LNetError: 328067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9841.506047] LNet: Removed LNI 192.168.204.136@tcp [ 9842.462670] Key type .llcrypt unregistered [ 9842.467818] Key type ._llcrypt unregistered [ 9867.718636] Key type ._llcrypt registered [ 9867.721226] Key type .llcrypt registered [ 9867.866427] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [ 9869.185231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9869.217230] alg: No test for adler32 (adler32-zlib) [ 9870.256560] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [ 9870.503216] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [ 9872.201406] Key type lgssc registered [ 9873.370609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9882.219711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9882.243563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9883.718412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9889.009657] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9893.866294] LustreError: 329488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9893.881749] LustreError: 329488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9898.979783] LustreError: 329487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9899.652143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9900.164386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9905.713287] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9913.796799] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9920.728496] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9925.090360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9925.096549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9925.104786] Lustre: Skipped 1 previous similar message [ 9925.131890] Lustre: Skipped 1 previous similar message [ 9926.115561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9926.129135] Lustre: Skipped 1 previous similar message [ 9930.833917] Lustre: server umount lustre-MDT0000 complete [ 9931.237895] LustreError: 329487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9931.256981] LustreError: 329487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9935.532847] LustreError: 329469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315745 with bad export cookie 13471281644570113626 [ 9935.542402] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9935.548089] LustreError: 329469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9935.908601] Lustre: server umount lustre-MDT0001 complete [ 9954.032491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9954.419592] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9954.493693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9954.811278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9960.846276] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9971.656531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9971.833036] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9971.894758] Lustre: 332075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9972.131268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9977.195547] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [ 9988.073635] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9996.517111] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10008.135851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10008.450050] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10008.466279] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10008.474783] LustreError: 333962:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10008.486669] LustreError: 333962:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10008.499048] LustreError: 333962:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10008.609357] Lustre: server umount lustre-OST0000 complete [10008.616189] LustreError: 333962:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10019.817921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10020.053342] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10020.263451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10026.187822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10026.897770] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10035.299234] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10044.441947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10044.593913] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10044.693374] Lustre: lustre-OST0001: new disk, initializing [10044.699181] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10044.707885] Lustre: 335504:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10044.821094] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10051.259669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10051.265988] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10051.405511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10052.126633] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10060.955826] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10066.928849] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10066.938515] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10066.940962] LustreError: Skipped 1 previous similar message [10066.947522] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10066.961267] Lustre: Skipped 1 previous similar message [10066.973935] Lustre: Skipped 1 previous similar message [10072.042789] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10072.060756] Lustre: Skipped 1 previous similar message [10077.160817] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10077.173294] Lustre: Skipped 1 previous similar message [10078.176155] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10078.356960] Lustre: server umount lustre-OST0001 complete [10082.277542] LustreError: 334427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10082.278273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10082.288154] LustreError: 334427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10082.300874] Lustre: Skipped 1 previous similar message [10087.397736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10087.401141] LustreError: 334428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10087.410548] Lustre: Skipped 3 previous similar messages [10087.437517] LustreError: 334428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10088.414642] Lustre: server umount lustre-OST0000 complete [10092.514989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10106.336197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10106.536235] Lustre: server umount lustre-MDT0000 complete [10107.873547] LustreError: 333228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10107.895054] LustreError: 333228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10110.272667] LustreError: 332066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315920 with bad export cookie 13471281644570114179 [10110.275263] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10110.286977] LustreError: 332066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10110.613339] Lustre: server umount lustre-MDT0001 complete [10122.317266] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [10125.408424] Key type lgssc unregistered [10125.680849] LNet: 337468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10125.688884] LNetError: 337468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10126.761482] LNet: Removed LNI 192.168.204.136@tcp [10127.582206] Key type .llcrypt unregistered [10127.585228] Key type ._llcrypt unregistered [10205.066906] Key type ._llcrypt registered [10205.070534] Key type .llcrypt registered [10205.161351] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [10223.329533] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [10225.078214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10225.144226] alg: No test for adler32 (adler32-zlib) [10226.761769] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [10227.198080] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [10228.968660] Key type lgssc registered [10230.720193] Lustre: Echo OBD driver; http://www.lustre.org/ [10270.654710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10270.700360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10271.993902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10272.053613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10272.136179] Lustre: lustre-MDT0000: new disk, initializing [10272.240805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10272.273930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10277.406227] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10289.114444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10289.264863] Lustre: 342408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10289.318458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10289.325117] Lustre: Skipped 1 previous similar message [10289.383239] Lustre: lustre-MDT0001: new disk, initializing [10289.502389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10289.576804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10289.606602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10293.987892] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10302.941671] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10309.950390] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10319.822738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10320.245470] Lustre: lustre-OST0000: new disk, initializing [10320.255499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10320.262853] Lustre: 344376:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10320.402979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10326.975694] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10329.712542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10329.719958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10329.862987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10336.162716] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10340.702939] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10340.899404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10345.560274] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10345.731644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10350.562886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10350.563468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10350.572825] LustreError: Skipped 1 previous similar message [10350.588882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10350.602553] Lustre: Skipped 1 previous similar message [10353.980459] Lustre: server umount lustre-OST0000 complete [10360.802667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10360.806253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10360.815879] Lustre: Skipped 2 previous similar messages [10364.051914] Lustre: server umount lustre-MDT0000 complete [10367.550496] LustreError: 342400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788316177 with bad export cookie 9185310196954332700 [10367.554272] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10367.560644] LustreError: 342400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10367.790270] Lustre: server umount lustre-MDT0001 complete [10379.475753] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:29:46 (1788316186) [10390.269332] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:29:57 (1788316197) [10400.539326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10401.359890] LustreError: 347129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.395241] LustreError: 347129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10401.516520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10406.881341] LustreError: 347130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10406.912903] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10412.004055] LustreError: 347129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10416.687782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10417.115086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10422.397934] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10431.690979] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10438.539652] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10446.909838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10447.457731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10454.442750] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10463.461871] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10481.697385] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10484.021431] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10485.745992] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10487.965401] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10490.396276] Lustre: Failing over lustre-MDT0000 [10490.693562] Lustre: server umount lustre-MDT0000 complete [10492.897888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10492.905167] LustreError: 349428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10492.916965] Lustre: Skipped 2 previous similar messages [10492.939805] LustreError: 349428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10498.020057] LustreError: 347126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10498.037090] LustreError: 347126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10508.257249] Lustre: 340632:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788316302/real 1788316302] req@00000000c05ab353 x1875185276886912/t0(0) o400->MGC192.168.204.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788316318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10508.261843] LustreError: 347859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10508.303221] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10508.317886] LustreError: 347859:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10511.014111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10518.500065] LustreError: 340628:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000092ec284 x1875185276891520/t0(0) o250->MGC192.168.204.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10518.887758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10518.968282] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10520.116024] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10523.973611] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10524.135592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10524.166832] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10524.201838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10536.790725] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10539.591561] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10544.317978] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10546.529891] Lustre: Failing over lustre-MDT0000 [10546.779324] Lustre: server umount lustre-MDT0000 complete [10549.738461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10549.751551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10549.757274] LustreError: 347124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10549.778513] Lustre: Skipped 2 previous similar messages [10549.800223] LustreError: 347124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10565.584622] Lustre: 340632:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788316359/real 1788316359] req@00000000e7d78d5e x1875185276908800/t0(0) o400->MGC192.168.204.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788316375 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10565.618929] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10565.908920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10576.149782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10576.225896] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10576.900864] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10581.492945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10581.503101] Lustre: Skipped 2 previous similar messages [10581.548035] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10581.618550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10581.663748] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10591.754768] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10593.358292] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10601.955167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10601.959830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10601.982236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10601.991314] Lustre: Skipped 1 previous similar message [10605.500270] Lustre: server umount lustre-MDT0000 complete [10607.073316] LustreError: 347859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10607.090098] LustreError: 347859:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 22 previous similar messages [10609.400769] LustreError: 347112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788316419 with bad export cookie 9185310196954335444 [10609.413039] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10609.414650] LustreError: 347112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10609.898896] Lustre: server umount lustre-MDT0001 complete [10624.063036] Lustre: server umount lustre-OST0000 complete [10641.519489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10642.124872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10647.752487] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10657.150549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10662.136098] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10671.290588] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10679.708927] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10688.535955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10694.591926] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10700.550434] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10711.414693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10717.667560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10717.679560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10717.688607] Lustre: Skipped 2 previous similar messages [10717.693726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10717.699193] Lustre: Skipped 2 previous similar messages [10721.761365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10721.767062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10721.788705] Lustre: Skipped 1 previous similar message [10723.373369] Lustre: server umount lustre-OST0000 complete [10732.002796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10732.008760] Lustre: Skipped 2 previous similar messages [10733.371745] Lustre: server umount lustre-MDT0000 complete [10737.122074] LustreError: 353497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10737.139435] LustreError: 353497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10737.668085] LustreError: 353483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788316547 with bad export cookie 9185310196954336165 [10737.675170] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10737.937979] Lustre: server umount lustre-MDT0001 complete [10750.508070] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [10754.088573] Key type lgssc unregistered [10754.356678] LNet: 357057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10754.368451] LNetError: 357057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10754.398851] LNet: Removed LNI 192.168.204.136@tcp [10755.433093] Key type .llcrypt unregistered [10755.437087] Key type ._llcrypt unregistered [10784.551915] Key type ._llcrypt registered [10784.555233] Key type .llcrypt registered [10784.671117] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [10785.679018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10785.704674] alg: No test for adler32 (adler32-zlib) [10786.764728] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [10787.101762] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [10788.792184] Key type lgssc registered [10789.871694] Lustre: Echo OBD driver; http://www.lustre.org/ [10799.097184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10799.151122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10801.089461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10807.119545] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10810.342251] LustreError: 358575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10810.381897] LustreError: 358575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10815.459982] LustreError: 358576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.252917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.730795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10823.572094] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10833.227514] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10839.786424] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10847.447628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10847.969178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10854.945145] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10864.252667] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10870.527240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10885.093094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10885.102061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10885.118260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10886.116250] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10886.118534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10886.157080] Lustre: Skipped 1 previous similar message [10886.187502] Lustre: Skipped 2 previous similar messages [10888.912448] Lustre: server umount lustre-MDT0000 complete [10891.241404] LustreError: 358576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10891.281948] LustreError: 358576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10893.030462] LustreError: 358557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788316702 with bad export cookie 6582163449315284490 [10893.036441] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10893.038963] LustreError: 358557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10893.276496] Lustre: server umount lustre-MDT0001 complete [10908.870662] Lustre: server umount lustre-OST0000 complete [10926.707054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10927.386430] LustreError: 362316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.507192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10932.485383] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10932.714135] LustreError: 362317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10941.602362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10942.089135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10946.775692] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10956.605884] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10965.108321] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10973.857552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10974.310054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10981.214978] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [10991.393164] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10997.884228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11018.735252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11018.737767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11018.739131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11018.766600] Lustre: Skipped 2 previous similar messages [11018.771490] Lustre: Skipped 1 previous similar message [11022.237282] Lustre: server umount lustre-MDT0000 complete [11023.845173] LustreError: 363892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11023.870095] LustreError: 363892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11027.227893] LustreError: 362297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788316837 with bad export cookie 6582163449315285603 [11027.229799] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11027.241913] LustreError: 362297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11027.816589] Lustre: server umount lustre-MDT0001 complete [11042.437868] Lustre: server umount lustre-OST0000 complete [11061.089981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11061.773844] LustreError: 366248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11061.897869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11068.383772] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11079.159176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11084.810054] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11094.536750] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11101.499991] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11109.659934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11110.148548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11110.156286] Lustre: Skipped 1 previous similar message [11116.863949] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11125.216519] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11131.713588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11142.123755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11142.136746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11142.147163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11142.154095] Lustre: Skipped 2 previous similar messages [11147.240491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11147.252911] Lustre: Skipped 1 previous similar message [11147.490850] Lustre: server umount lustre-OST0000 complete [11151.339551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11151.344370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11151.355984] Lustre: Skipped 1 previous similar message [11157.473799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11157.498055] Lustre: Skipped 4 previous similar messages [11157.701872] Lustre: server umount lustre-MDT0000 complete [11161.558210] LustreError: 366228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788316971 with bad export cookie 6582163449315286625 [11161.570729] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11161.582656] LustreError: 366228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11162.029434] Lustre: server umount lustre-MDT0001 complete [11176.406190] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [11179.397912] Key type lgssc unregistered [11179.681830] LNet: 369808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11179.692332] LNetError: 369808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11179.719613] LNet: Removed LNI 192.168.204.136@tcp [11180.601343] Key type .llcrypt unregistered [11180.604740] Key type ._llcrypt unregistered [11207.831963] Key type ._llcrypt registered [11207.837541] Key type .llcrypt registered [11207.961290] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [11209.576558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11209.625046] alg: No test for adler32 (adler32-zlib) [11210.716373] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [11211.104050] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [11212.792183] Key type lgssc registered [11214.248832] Lustre: Echo OBD driver; http://www.lustre.org/ [11224.404527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11224.464428] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.491378] LustreError: 371251:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11224.520723] LustreError: 371251:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11224.532813] LustreError: 371251:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11224.543284] LustreError: 371251:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11224.555223] LustreError: 371251:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11233.617711] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11233.630194] LustreError: 371649:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11233.661403] LustreError: 371649:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11233.669437] LustreError: 371649:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11233.681331] LustreError: 371649:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11233.689539] LustreError: 371649:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11256.603695] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [11259.411035] Key type lgssc unregistered [11259.709779] LNet: 372656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11259.714948] LNetError: 372656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11259.729931] LNet: Removed LNI 192.168.204.136@tcp [11260.666238] Key type .llcrypt unregistered [11260.669506] Key type ._llcrypt unregistered [11281.104877] Key type ._llcrypt registered [11281.107918] Key type .llcrypt registered [11281.238234] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [11299.800282] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [11301.190141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11301.244324] alg: No test for adler32 (adler32-zlib) [11302.274704] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [11302.593878] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [11304.289373] Key type lgssc registered [11305.868658] Lustre: Echo OBD driver; http://www.lustre.org/ [11349.055435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11349.082143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11350.373225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11350.448551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11350.620572] Lustre: lustre-MDT0000: new disk, initializing [11350.733634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11350.774050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11356.844663] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11369.094502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11369.281646] Lustre: 375817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11369.344964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11369.348291] Lustre: Skipped 1 previous similar message [11369.424742] Lustre: lustre-MDT0001: new disk, initializing [11369.514932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11369.560063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11369.574882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11374.667536] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11385.561821] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11392.692234] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11400.891369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11401.203419] Lustre: lustre-OST0000: new disk, initializing [11401.211200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11401.220184] Lustre: 377784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11401.351059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11406.890147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11406.907811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11407.032042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11408.012713] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11419.311667] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11424.599576] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11424.820462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11429.700194] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11430.005904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11432.931625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11432.932300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11432.949607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11438.053098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11438.070028] Lustre: Skipped 2 previous similar messages [11443.172590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11443.182249] Lustre: Skipped 1 previous similar message [11446.240229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11446.465132] Lustre: server umount lustre-OST0000 complete [11451.365418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11451.386975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11451.413691] Lustre: Skipped 1 previous similar message [11451.424152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11453.414622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11458.534334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11458.545692] Lustre: Skipped 3 previous similar messages [11465.184173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11465.420121] Lustre: server umount lustre-MDT0000 complete [11468.884314] LustreError: 375806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788317278 with bad export cookie 12504066930069944138 [11468.887443] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11468.894630] LustreError: 375806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11469.314055] Lustre: server umount lustre-MDT0001 complete [11478.670722] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11480.420806] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:48:08 (1788317288) [11487.810783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11488.386718] LustreError: 380249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11488.415134] LustreError: 380249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11488.474401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11492.759529] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11493.869080] LustreError: 380250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11499.009642] LustreError: 380249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11502.455495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11502.820353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11508.682631] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11518.222078] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11525.870172] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11531.919934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11532.308422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11538.160617] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11546.657556] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11555.347358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11555.627410] Lustre: lustre-OST0001: new disk, initializing [11555.630435] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11555.634601] Lustre: 383085:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11555.707172] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11563.358465] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11565.669221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11565.681534] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11565.752297] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11574.674852] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11581.412180] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11581.414441] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11581.421373] LustreError: Skipped 1 previous similar message [11581.427054] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11581.427062] Lustre: Skipped 2 previous similar messages [11581.465398] Lustre: Skipped 1 previous similar message [11584.999644] Lustre: server umount lustre-OST0001 complete [11586.531983] LustreError: 382072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11586.569835] LustreError: 382072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11591.649637] LustreError: 382071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11591.678150] LustreError: 382071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11601.909573] LustreError: 382071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11601.944137] LustreError: 382071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11603.940095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11603.948936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11603.981286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11603.996471] Lustre: Skipped 2 previous similar messages [11617.250055] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11617.568720] Lustre: server umount lustre-OST0000 complete [11622.373574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11622.385760] Lustre: Skipped 1 previous similar message [11628.004598] Lustre: server umount lustre-MDT0000 complete [11631.326038] LustreError: 383084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788317441 with bad export cookie 12504066930069945020 [11631.335120] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11637.650711] Lustre: server umount lustre-MDT0001 complete [11648.848668] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [11651.664591] Key type lgssc unregistered [11651.889744] LNet: 385082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11651.901960] LNetError: 385082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11651.927699] LNet: Removed LNI 192.168.204.136@tcp [11652.728538] Key type .llcrypt unregistered [11652.730281] Key type ._llcrypt unregistered [11708.141179] Key type ._llcrypt registered [11708.148372] Key type .llcrypt registered [11708.256796] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [11709.506462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11709.558407] alg: No test for adler32 (adler32-zlib) [11710.878233] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [11711.155043] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [11712.865895] Key type lgssc registered [11714.501177] Lustre: Echo OBD driver; http://www.lustre.org/ [11722.559370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11722.574474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11723.783897] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11723.806867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11723.961545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11729.724640] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11744.277996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11744.339932] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11744.367188] Lustre: 387884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11744.591660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11751.582395] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11759.421627] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11766.374943] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11770.339496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11770.350096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11770.364246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11772.388109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11772.397722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11772.434474] Lustre: Skipped 1 previous similar message [11777.510755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11777.517177] Lustre: Skipped 1 previous similar message [11782.626985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11782.644465] Lustre: Skipped 1 previous similar message [11783.136134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11783.369990] Lustre: server umount lustre-MDT0000 complete [11787.327945] LustreError: 387873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788317597 with bad export cookie 9488729004692564017 [11787.347465] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11787.356645] LustreError: 387873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11787.966730] Lustre: server umount lustre-MDT0001 complete [11800.058569] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:53:27 (1788317607) [11809.349919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11810.245362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11815.403695] LustreError: 390400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11815.441918] LustreError: 390400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11816.801186] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11820.529376] LustreError: 390399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11825.645789] LustreError: 390400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11826.410909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11826.807369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11832.014798] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11841.317652] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11847.759587] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11855.135962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11855.413756] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11855.680338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11862.737735] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11870.123777] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11878.605955] Lustre: Permanently deactivating lustre-OST0000 [11878.646834] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11888.676470] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11890.944123] LustreError: 393436:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11890.961668] LustreError: 393436:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11890.974062] LustreError: 393436:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11894.363266] LustreError: 393484:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11894.371567] LustreError: 393484:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11902.964636] Lustre: server umount lustre-OST0000 complete [11906.805932] LustreError: 393736:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11906.825979] LustreError: 393736:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11911.138563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11911.148350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11911.168629] Lustre: Skipped 1 previous similar message [11919.435460] Lustre: server umount lustre-MDT0000 complete [11921.390289] LustreError: 390400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11921.442071] LustreError: 390400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11925.323759] LustreError: 390382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788317735 with bad export cookie 9488729004692564598 [11925.336982] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11925.341921] LustreError: 390382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11935.964574] Lustre: server umount lustre-MDT0001 complete [11946.980911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11952.893545] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11956.726907] LustreError: 394896:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11964.140369] Lustre: 395235:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.136@tcp [11966.397311] Lustre: server umount MGS complete [11979.834585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11980.510816] LustreError: 395959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.546626] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11980.555986] Lustre: Skipped 1 previous similar message [11980.609453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11986.962348] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [11991.012795] LustreError: 395960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11991.030031] LustreError: 395960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11997.159601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11997.481997] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11997.567432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12004.165666] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12012.797613] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12019.391690] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12021.267102] Lustre: Permanently reactivating lustre-OST0000 [12021.284443] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12021.288692] Lustre: Skipped 2 previous similar messages [12051.936367] LustreError: 397791:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12061.154764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12061.682895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12070.058284] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12080.224221] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12092.392320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12092.395957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12092.408326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12092.408338] Lustre: Skipped 3 previous similar messages [12092.431817] LustreError: Skipped 1 previous similar message [12093.813645] Lustre: server umount lustre-OST0000 complete [12100.076739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12100.091863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12100.106109] Lustre: Skipped 1 previous similar message [12112.352156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12112.576122] Lustre: server umount lustre-MDT0000 complete [12112.870391] LustreError: 395959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12112.896759] LustreError: 395959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12117.438565] LustreError: 395940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788317927 with bad export cookie 9488729004692565648 [12117.447933] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12117.454301] LustreError: 395940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12117.982572] Lustre: server umount lustre-MDT0001 complete [12130.512385] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [12133.970868] Key type lgssc unregistered [12134.304943] LNet: 400379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12134.314210] LNetError: 400379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12134.331665] LNet: Removed LNI 192.168.204.136@tcp [12135.154948] Key type .llcrypt unregistered [12135.157492] Key type ._llcrypt unregistered [12156.847633] Key type ._llcrypt registered [12156.852404] Key type .llcrypt registered [12157.024288] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [12174.463593] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [12175.566680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12175.590201] alg: No test for adler32 (adler32-zlib) [12176.726991] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [12176.932991] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [12178.608945] Key type lgssc registered [12180.048928] Lustre: Echo OBD driver; http://www.lustre.org/ [12222.123383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12222.171567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12223.658644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12223.731090] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12223.877266] Lustre: lustre-MDT0000: new disk, initializing [12224.004965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12224.034658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12229.793705] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12243.467324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12243.688653] Lustre: 403537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12243.755716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12243.760585] Lustre: Skipped 1 previous similar message [12243.900782] Lustre: lustre-MDT0001: new disk, initializing [12244.113263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12244.158071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12244.171315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12250.031973] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12260.690925] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12267.151646] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12275.403298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12275.922806] Lustre: lustre-OST0000: new disk, initializing [12275.926652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12275.932780] Lustre: 405519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12276.073147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12283.939565] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12286.044541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12286.055614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12286.200791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12295.033647] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12301.315168] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12301.607115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12307.337666] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12307.570363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12312.045376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12312.053809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12312.084661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12312.090806] Lustre: Skipped 1 previous similar message [12316.378768] Lustre: server umount lustre-OST0000 complete [12322.277120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12322.281345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12322.289343] Lustre: Skipped 2 previous similar messages [12322.301824] Lustre: Skipped 1 previous similar message [12327.362120] Lustre: server umount lustre-MDT0000 complete [12331.519815] LustreError: 403530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788318141 with bad export cookie 16934323538503968893 [12331.527257] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12331.538234] LustreError: 403530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12331.856713] Lustre: server umount lustre-MDT0001 complete [12340.865900] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12342.861801] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 23:02:30 (1788318150) [12351.298493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12351.950897] LustreError: 407977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12351.974893] LustreError: 407977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12352.073065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12356.493734] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12357.098900] LustreError: 407978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12357.132231] LustreError: 407978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12361.195972] LustreError: 407977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12366.307116] LustreError: 407978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12366.684320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12367.048166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12372.112494] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12379.096144] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12385.309686] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12392.810601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12393.263108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12400.225228] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12408.978653] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12411.115607] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12414.917554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12422.116319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12422.117272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12422.125479] LustreError: Skipped 2 previous similar messages [12422.127041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12422.158208] Lustre: Skipped 1 previous similar message [12426.536594] Lustre: server umount lustre-OST0000 complete [12432.356752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12432.369783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12432.378277] Lustre: Skipped 2 previous similar messages [12437.482154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12437.496462] Lustre: Skipped 1 previous similar message [12445.153804] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12445.562752] Lustre: server umount lustre-MDT0000 complete [12447.717633] LustreError: 410419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12447.739169] LustreError: 410419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12449.758389] LustreError: 407958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788318259 with bad export cookie 16934323538503969775 [12449.763182] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12449.773308] LustreError: 407958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12450.454733] Lustre: server umount lustre-MDT0001 complete [12460.904942] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [12463.677761] Key type lgssc unregistered [12463.955595] LNet: 411708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12463.962415] LNetError: 411708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12463.978273] LNet: Removed LNI 192.168.204.136@tcp [12464.777950] Key type .llcrypt unregistered [12464.786435] Key type ._llcrypt unregistered [12490.721563] Key type ._llcrypt registered [12490.726320] Key type .llcrypt registered [12490.906108] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [12491.383414] Key type .llcrypt unregistered [12491.386859] Key type ._llcrypt unregistered [12509.584793] Key type ._llcrypt registered [12509.589144] Key type .llcrypt registered [12509.729983] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [12510.709945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12510.746684] alg: No test for adler32 (adler32-zlib) [12511.779355] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [12512.018844] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [12513.704286] Key type lgssc registered [12515.006604] Lustre: Echo OBD driver; http://www.lustre.org/ [12524.014494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12524.065467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12525.680417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12531.244048] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12535.277787] LustreError: 413929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12535.302779] LustreError: 413929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12539.412810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12539.817339] LustreError: 413928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12539.852115] LustreError: 413928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12539.862591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12544.995297] LustreError: 413929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12545.006860] LustreError: 413929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12546.690069] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12550.114767] LustreError: 413925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12550.135044] LustreError: 413925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12551.209032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12556.132453] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12556.264337] LustreError: 413923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12556.282709] LustreError: 413923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12563.830127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12564.229053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12568.894601] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12579.052371] Lustre: server umount lustre-OST0000 complete [12584.932500] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12584.932613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12584.936961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12584.943993] LustreError: Skipped 1 previous similar message [12584.952458] Lustre: Skipped 1 previous similar message [12590.051428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12590.072167] Lustre: Skipped 2 previous similar messages [12593.214795] Lustre: server umount lustre-MDT0000 complete [12595.169583] Lustre: 413240:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788318389/real 1788318389] req@00000000efc7df8f x1875187671788928/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788318405 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12595.178828] LustreError: 413928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12595.214322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12595.240501] LustreError: 413928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12596.192612] Lustre: 413241:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788318389/real 1788318389] req@00000000fb586f3d x1875187671788544/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788318405 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12597.685267] LustreError: 413910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788318407 with bad export cookie 11115704297833131325 [12597.693823] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12597.702156] LustreError: 413910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12597.990374] Lustre: server umount lustre-MDT0001 complete [12610.789910] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [12614.587567] Key type lgssc unregistered [12614.921988] LNet: 417081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12614.933763] LNetError: 417081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12615.977852] LNet: Removed LNI 192.168.204.136@tcp [12617.085594] Key type .llcrypt unregistered [12617.088548] Key type ._llcrypt unregistered [12644.693444] Key type ._llcrypt registered [12644.696313] Key type .llcrypt registered [12644.841596] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [12646.307439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12646.414229] alg: No test for adler32 (adler32-zlib) [12647.635390] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [12648.030239] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [12649.752313] Key type lgssc registered [12651.051507] Lustre: Echo OBD driver; http://www.lustre.org/ [12660.465542] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12660.498560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12676.071052] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12681.184374] LustreError: 418474:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.136@tcp: failed processing log, type 4: rc = -110 [12707.872560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12714.144831] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12722.122411] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12729.280269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12729.406575] LustreError: 417810:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f0d42c53 x1875187814369536/t0(0) o250->MGC192.168.204.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12729.860666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12729.964310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12734.958071] LustreError: 418499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12734.984545] LustreError: 418499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12735.089554] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12740.069080] LustreError: 418500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12743.155271] LustreError: 418499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12743.184272] LustreError: 418499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12744.487514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12744.916942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12749.984239] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12758.986882] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12765.104030] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12775.240721] Lustre: server umount lustre-OST0000 complete [12779.938766] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12779.943773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12779.957772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12781.026131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12781.027935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12786.155672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12786.175815] Lustre: Skipped 2 previous similar messages [12791.264544] Lustre: 417814:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788318585/real 1788318585] req@00000000e80b199d x1875187814398720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788318601 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12791.267234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12791.270656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12791.307653] Lustre: 417814:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12794.337439] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12794.496705] Lustre: server umount lustre-MDT0000 complete [12796.385101] Lustre: 417812:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788318590/real 1788318590] req@00000000de654546 x1875187814400768/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788318606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12796.386236] LustreError: 420518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12796.412804] LustreError: 420518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12797.838985] LustreError: 418481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788318607 with bad export cookie 876880103967458777 [12797.845509] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12798.190310] Lustre: server umount lustre-MDT0001 complete [12812.680540] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [12816.047867] Key type lgssc unregistered [12816.358246] LNet: 422049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12816.376138] LNetError: 422049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12817.448633] LNet: Removed LNI 192.168.204.136@tcp [12818.474412] Key type .llcrypt unregistered [12818.476979] Key type ._llcrypt unregistered [12844.679886] Key type ._llcrypt registered [12844.682192] Key type .llcrypt registered [12844.736168] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [12845.479478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12845.512237] alg: No test for adler32 (adler32-zlib) [12846.443362] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [12846.658973] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [12848.368220] Key type lgssc registered [12849.435363] Lustre: Echo OBD driver; http://www.lustre.org/ [12858.125496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12858.135916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12859.661775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12865.493247] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12870.120747] LustreError: 423466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12870.146083] LustreError: 423466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12875.067230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12875.234636] LustreError: 423467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12875.516779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12880.456701] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12889.046255] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12896.070208] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12904.021925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12904.364175] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [12904.494701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12909.569156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12911.691393] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [12920.622755] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12925.930867] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12925.941066] Lustre: Skipped 1 previous similar message [12925.942652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12925.948793] Lustre: Skipped 1 previous similar message [12927.525230] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.36@tcp (stopping) [12927.530303] Lustre: Skipped 1 previous similar message [12930.595247] Lustre: server umount lustre-MDT0000 complete [12931.043690] LustreError: 424209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12931.064634] LustreError: 424209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12933.679216] LustreError: 423447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.36@tcp arrived at 1788318743 with bad export cookie 16575887734355074630 [12936.115811] LustreError: 423446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788318746 with bad export cookie 16575887734355073734 [12936.132693] LustreError: 423446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12936.140590] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12936.162181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12936.169851] LustreError: 423466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12936.172190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12936.172454] Lustre: Skipped 1 previous similar message [12936.174533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12936.209276] LustreError: 423466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12941.283346] LustreError: 423467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12941.287227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12941.312836] LustreError: 423467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12942.432776] Lustre: server umount lustre-OST0000 complete [12950.190180] Lustre: server umount lustre-MDT0001 complete [12963.457582] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [12967.032779] Key type lgssc unregistered [12967.422897] LNet: 427157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12967.429444] LNetError: 427157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12967.468474] LNet: Removed LNI 192.168.204.136@tcp [12968.767212] Key type .llcrypt unregistered [12968.769165] Key type ._llcrypt unregistered [12998.148317] Key type ._llcrypt registered [12998.150473] Key type .llcrypt registered [12998.255739] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [12999.620432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12999.673685] alg: No test for adler32 (adler32-zlib) [13000.829658] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [13001.140968] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [13002.880853] Key type lgssc registered [13004.237792] Lustre: Echo OBD driver; http://www.lustre.org/ [13013.360575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13013.372949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13014.838717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13019.449342] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13024.230036] LustreError: 428575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13024.253041] LustreError: 428575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13029.079117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13029.350782] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13029.649074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13034.956053] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13043.469513] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13049.826203] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13057.851047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13058.306713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13059.353448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13064.979501] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13075.563482] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13081.065835] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13081.087569] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13081.141497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13082.658138] LustreError: 428572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13082.930922] Lustre: server umount lustre-MDT0001 complete [13085.162373] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13085.163820] LustreError: 429315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13085.205236] LustreError: 429315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13090.277610] LustreError: 428576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13090.279916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13090.309319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13090.326828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13095.452543] LustreError: 429315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13095.494420] LustreError: 429315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13096.134619] Lustre: server umount lustre-OST0000 complete [13102.183149] Lustre: server umount lustre-MDT0000 complete [13120.940169] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [13124.363522] Key type lgssc unregistered [13124.694143] LNet: 432265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13124.714874] LNetError: 432265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13124.738127] LNet: Removed LNI 192.168.204.136@tcp [13125.846413] Key type .llcrypt unregistered [13125.848696] Key type ._llcrypt unregistered [13155.657246] Key type ._llcrypt registered [13155.661381] Key type .llcrypt registered [13155.849354] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [13171.381590] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [13173.004493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13173.168245] alg: No test for adler32 (adler32-zlib) [13174.333396] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [13174.725791] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [13176.400313] Key type lgssc registered [13178.084179] Lustre: Echo OBD driver; http://www.lustre.org/ [13222.592208] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [13239.585297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13239.644894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13240.916424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13240.937189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13241.011193] Lustre: lustre-MDT0000: new disk, initializing [13241.097962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13241.120868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13244.999771] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13262.460740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13262.986781] Lustre: lustre-OST0000: new disk, initializing [13262.992576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13262.997502] Lustre: Skipped 1 previous similar message [13263.004293] Lustre: 437300:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13263.108556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13271.633674] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13272.877612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13272.908080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13272.957869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13282.585445] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13297.338612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13297.481220] Lustre: 436282:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13297.550230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13297.631809] Lustre: lustre-MDT0001: new disk, initializing [13297.742219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13297.776337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13297.788264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13301.292148] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13310.339961] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13310.583437] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13314.820658] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13315.013414] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13320.236476] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [13323.251261] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13323.260969] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13323.266527] Lustre: Skipped 1 previous similar message [13323.292436] Lustre: Skipped 1 previous similar message [13325.313396] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.36@tcp (stopping) [13325.661053] Lustre: server umount lustre-MDT0001 complete [13333.480088] LustreError: 436294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13333.498447] LustreError: 436294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13338.612793] LustreError: 436288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13338.645490] LustreError: 436288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13343.716129] LustreError: 436293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13343.719404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13343.758155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13348.833975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13348.834689] LustreError: 436294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13349.079341] Lustre: server umount lustre-OST0000 complete [13367.267933] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13367.690395] Lustre: server umount lustre-MDT0000 complete [13381.517553] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [13384.520397] Key type lgssc unregistered [13384.803705] LNet: 441053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13384.812213] LNetError: 441053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13384.826857] LNet: Removed LNI 192.168.204.136@tcp [13385.651898] Key type .llcrypt unregistered [13385.655973] Key type ._llcrypt unregistered [13422.337062] Key type ._llcrypt registered [13422.338765] Key type .llcrypt registered [13422.467736] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [13422.782325] Key type .llcrypt unregistered [13422.785540] Key type ._llcrypt unregistered [13439.618017] Key type ._llcrypt registered [13439.621175] Key type .llcrypt registered [13439.701912] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [13440.476736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13440.490495] alg: No test for adler32 (adler32-zlib) [13441.459576] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [13441.647791] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [13443.345436] Key type lgssc registered [13444.415335] Lustre: Echo OBD driver; http://www.lustre.org/ [13477.135499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13477.187880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13478.571497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13478.629907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13478.821349] Lustre: lustre-MDT0000: new disk, initializing [13479.013630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13479.036624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13484.309871] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13496.435442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13496.557979] Lustre: 444436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13496.632934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13496.637617] Lustre: Skipped 1 previous similar message [13496.717551] Lustre: lustre-MDT0001: new disk, initializing [13496.870755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13496.910742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13496.921349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13502.225141] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13513.144334] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13519.782192] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13527.655537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13528.101749] Lustre: lustre-OST0000: new disk, initializing [13528.111049] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13528.123531] Lustre: 446405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13528.288322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13535.712843] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13537.307129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13537.323930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13537.436382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13545.356828] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13553.125550] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13553.128661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13553.133070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13553.140015] Lustre: Skipped 2 previous similar messages [13558.223699] Lustre: server umount lustre-MDT0000 complete [13558.245504] LustreError: 445661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13558.264718] LustreError: 445661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13562.623251] LustreError: 444427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788319372 with bad export cookie 14295897359024915600 [13562.640227] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13562.648074] LustreError: 444427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13563.278953] Lustre: server umount lustre-MDT0001 complete [13578.328650] Lustre: server umount lustre-OST0000 complete [13590.494619] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:23:17 (1788319397) [13599.628299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13600.279722] LustreError: 448435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13600.387638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13605.850897] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13605.865920] LustreError: 448436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13610.977201] LustreError: 448435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13616.100784] LustreError: 448436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13616.611969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13617.151804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13622.804297] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13631.662760] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13637.716140] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13657.648106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13658.078632] Lustre: Found index 0 for lustre-OST0000, updating log [13658.352613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13664.875586] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [13671.746136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13673.737609] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13682.158778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13682.166818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13682.191381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13682.198153] LustreError: Skipped 1 previous similar message [13682.215509] Lustre: Skipped 3 previous similar messages [13683.245506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13683.255416] Lustre: Skipped 1 previous similar message [13686.855856] Lustre: server umount lustre-OST0000 complete [13704.173142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13704.183350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13704.198340] Lustre: Skipped 1 previous similar message [13704.211495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13709.629456] Lustre: server umount lustre-MDT0000 complete [13709.796934] LustreError: 448436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13714.120545] LustreError: 450279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788319524 with bad export cookie 14295897359024917000 [13714.131088] LustreError: 450279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13714.137712] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13714.404798] Lustre: server umount lustre-MDT0001 complete [13717.152206] LNet: 1 local NIs in recovery (showing 1): 192.168.204.136@tcp [13725.165967] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:25:32 (1788319532) [14353.254754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14354.027753] LustreError: 454854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14354.052894] LustreError: 454854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [14354.137638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14359.301676] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14359.526522] LustreError: 454855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14363.629159] LustreError: 454854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14367.993638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14368.448233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14372.727644] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14381.122539] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14386.594914] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14406.581480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14406.770886] Lustre: Found index 0 for lustre-OST0000, updating log [14407.105582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14408.363277] LNet: 1 local NIs in recovery (showing 1): 192.168.204.136@tcp [14414.424122] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14423.737915] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14426.593693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14426.618536] Lustre: Skipped 1 previous similar message [14426.636308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14426.640395] Lustre: Skipped 1 previous similar message [14432.348721] Lustre: server umount lustre-OST0000 complete [14437.157309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14437.163840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14437.177104] Lustre: Skipped 1 previous similar message [14437.195032] Lustre: Skipped 1 previous similar message [14438.250730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14438.257580] Lustre: Skipped 1 previous similar message [14440.291233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14440.305734] Lustre: Skipped 3 previous similar messages [14442.552590] Lustre: server umount lustre-MDT0000 complete [14443.364741] LustreError: 454855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14443.389161] LustreError: 454855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14446.749419] LustreError: 456715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788320256 with bad export cookie 14295897359024918085 [14446.758711] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14446.763556] LustreError: 456715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14447.167040] Lustre: server umount lustre-MDT0001 complete [14468.834176] LNet: 1 local NIs in recovery (showing 1): 192.168.204.136@tcp [14480.791604] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [14491.399906] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [14529.294872] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:38:56 (1788320336) [14548.818280] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [14552.237922] Key type lgssc unregistered [14552.649887] LNet: 461470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14552.664493] LNetError: 461470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14669.414836] LNet: Removed LNI 192.168.204.136@tcp [14670.417581] Key type .llcrypt unregistered [14670.419498] Key type ._llcrypt unregistered [14688.652599] Key type ._llcrypt registered [14688.657748] Key type .llcrypt registered [14688.857786] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [14690.178553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14690.211039] alg: No test for adler32 (adler32-zlib) [14691.597148] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [14691.931693] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [14693.584880] Key type lgssc registered [14695.059268] Lustre: Echo OBD driver; http://www.lustre.org/ [14699.964270] LNet: Added LNI 192.168.204.136@tcp100 [8/256/0/180] [14701.147444] LNet: Added LNI 192.168.204.136@tcp101 [8/256/0/180] [14702.188793] LNet: Added LNI 192.168.204.136@tcp102 [8/256/0/180] [14703.175355] LNet: Added LNI 192.168.204.136@tcp103 [8/256/0/180] [14704.184621] LNet: Added LNI 192.168.204.136@tcp104 [8/256/0/180] [14705.146311] LNet: Added LNI 192.168.204.136@tcp105 [8/256/0/180] [14706.125111] LNet: Added LNI 192.168.204.136@tcp106 [8/256/0/180] [14707.162845] LNet: Added LNI 192.168.204.136@tcp107 [8/256/0/180] [14708.200492] LNet: Added LNI 192.168.204.136@tcp108 [8/256/0/180] [14709.220559] LNet: Added LNI 192.168.204.136@tcp109 [8/256/0/180] [14710.293962] LNet: Added LNI 192.168.204.136@tcp110 [8/256/0/180] [14711.411337] LNet: Added LNI 192.168.204.136@tcp111 [8/256/0/180] [14712.436044] LNet: Added LNI 192.168.204.136@tcp112 [8/256/0/180] [14713.496512] LNet: Added LNI 192.168.204.136@tcp113 [8/256/0/180] [14714.595585] LNet: Added LNI 192.168.204.136@tcp114 [8/256/0/180] [14715.738697] LNet: Added LNI 192.168.204.136@tcp115 [8/256/0/180] [14716.787934] LNet: Added LNI 192.168.204.136@tcp116 [8/256/0/180] [14717.859089] LNet: Added LNI 192.168.204.136@tcp117 [8/256/0/180] [14718.826576] LNet: Added LNI 192.168.204.136@tcp118 [8/256/0/180] [14719.739348] LNet: Added LNI 192.168.204.136@tcp119 [8/256/0/180] [14720.780834] LNet: Added LNI 192.168.204.136@tcp120 [8/256/0/180] [14721.932198] LNet: Added LNI 192.168.204.136@tcp121 [8/256/0/180] [14723.097843] LNet: Added LNI 192.168.204.136@tcp122 [8/256/0/180] [14724.099493] LNet: Added LNI 192.168.204.136@tcp123 [8/256/0/180] [14725.114642] LNet: Added LNI 192.168.204.136@tcp124 [8/256/0/180] [14726.180109] LNet: Added LNI 192.168.204.136@tcp125 [8/256/0/180] [14727.073854] LNet: Added LNI 192.168.204.136@tcp126 [8/256/0/180] [14728.034710] LNet: Added LNI 192.168.204.136@tcp127 [8/256/0/180] [14729.041992] LNet: Added LNI 192.168.204.136@tcp128 [8/256/0/180] [14729.968413] LNet: Added LNI 192.168.204.136@tcp129 [8/256/0/180] [14730.867716] LNet: Added LNI 192.168.204.136@tcp130 [8/256/0/180] [14731.721437] LNet: Added LNI 192.168.204.136@tcp131 [8/256/0/180] [14732.478034] LNet: Added LNI 192.168.204.136@tcp132 [8/256/0/180] [14733.423520] LNet: Added LNI 192.168.204.136@tcp133 [8/256/0/180] [14734.374617] LNet: Added LNI 192.168.204.136@tcp134 [8/256/0/180] [14735.334732] LNet: Added LNI 192.168.204.136@tcp135 [8/256/0/180] [14742.749863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14742.787125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14743.938075] Lustre: 464502:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [14743.970843] LNetError: 464522:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14743.996229] LNetError: Protocol error connecting to 192.168.204.136@tcp115 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14744.012487] LNetError: Skipped 9 previous similar messages [14744.091523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14744.110805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14744.165710] LNet: 4 local NIs in recovery (showing 4): 192.168.204.136@tcp126, 192.168.204.136@tcp127, 192.168.204.136@tcp129, 192.168.204.136@tcp128 [14744.181621] Lustre: lustre-MDT0000: new disk, initializing [14744.285695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14744.307652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14746.274770] LNetError: 464522:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14746.305513] LNetError: 464522:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 17 previous similar messages [14746.317390] LNetError: Protocol error connecting to 192.168.204.136@tcp127 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14746.337128] LNetError: Skipped 8 previous similar messages [14748.803391] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14750.386737] LNetError: 464525:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14750.413255] LNetError: Protocol error connecting to 192.168.204.136@tcp127 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14758.567525] LNetError: 461961:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14758.576762] LNetError: Protocol error connecting to 192.168.204.136@tcp127 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14760.096937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14760.297559] Lustre: 464519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14760.411257] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14760.417919] Lustre: Skipped 1 previous similar message [14760.545040] Lustre: lustre-MDT0001: new disk, initializing [14760.578711] Lustre: 465322:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [14760.586044] Lustre: 465322:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14760.693948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14760.743438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14760.754501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14765.390315] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14768.191550] Lustre: 465686:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [14768.207914] Lustre: 465686:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14773.857811] LNetError: 464523:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14773.878905] LNetError: Protocol error connecting to 192.168.204.136@tcp107 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14775.646637] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14780.987547] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14782.060677] LNetError: 464529:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14782.070560] LNetError: 464529:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14782.078313] LNetError: Protocol error connecting to 192.168.204.136@tcp122 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14782.085295] LNetError: Skipped 7 previous similar messages [14786.712400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14786.947455] Lustre: lustre-OST0000: new disk, initializing [14786.956531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14786.963231] Lustre: 466496:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14787.080362] Lustre: 466501:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [14787.090473] Lustre: 466501:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14787.132852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14792.733871] Lustre: 466782:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.136@tcp: can't add NID 192.168.204.136@tcp130: rc = -75 [14792.740683] Lustre: 466782:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14792.813926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14792.835186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14792.986157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14794.164423] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [14798.435879] LNetError: 461962:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14798.447837] LNetError: 461962:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14798.456514] LNetError: Protocol error connecting to 192.168.204.136@tcp107 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14798.465443] LNetError: Skipped 10 previous similar messages [14803.965742] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14804.640894] LNet: 1 local NIs in recovery (showing 1): 192.168.204.136@tcp127 [14813.675953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14813.691193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14813.696922] LustreError: Skipped 1 previous similar message [14813.699372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14813.734706] Lustre: Skipped 1 previous similar message [14818.095893] Lustre: server umount lustre-OST0000 complete [14823.905725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14823.921366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14823.922310] Lustre: Skipped 1 previous similar message [14823.944279] Lustre: Skipped 2 previous similar messages [14829.092209] Lustre: server umount lustre-MDT0000 complete [14833.017644] LustreError: 464510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788320642 with bad export cookie 1002565255380406412 [14833.020050] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14833.027442] LustreError: 464510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14833.508971] Lustre: server umount lustre-MDT0001 complete [14840.421420] LNetError: 464529:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.136 [14840.433901] LNetError: 464529:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [14840.440445] LNetError: Protocol error connecting to 192.168.204.136@tcp113 at host 192.168.204.136:988: is it running a compatible version of Lustre? [14840.448517] LNetError: Skipped 18 previous similar messages [14844.662233] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [14847.658599] Key type lgssc unregistered [14848.147903] LNet: 468413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14848.165893] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14848.173396] LNet: Removed LNI 192.168.204.136@tcp [14849.184129] LNet: Removed LNI 192.168.204.136@tcp100 [14849.190954] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14849.204207] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14849.210931] LNet: Removed LNI 192.168.204.136@tcp101 [14850.274611] LNet: Removed LNI 192.168.204.136@tcp102 [14850.298911] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14850.309204] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14850.317917] LNet: Removed LNI 192.168.204.136@tcp103 [14851.361978] LNet: Removed LNI 192.168.204.136@tcp104 [14852.384159] LNet: Removed LNI 192.168.204.136@tcp105 [14852.388632] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14852.398863] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14852.408147] LNet: Removed LNI 192.168.204.136@tcp106 [14853.474347] LNet: Removed LNI 192.168.204.136@tcp107 [14854.496228] LNet: Removed LNI 192.168.204.136@tcp108 [14855.521351] LNet: Removed LNI 192.168.204.136@tcp109 [14856.544190] LNet: Removed LNI 192.168.204.136@tcp110 [14856.550391] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14856.558461] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14856.572438] LNet: Removed LNI 192.168.204.136@tcp111 [14857.632194] LNet: Removed LNI 192.168.204.136@tcp112 [14858.656508] LNet: Removed LNI 192.168.204.136@tcp113 [14859.680868] LNet: Removed LNI 192.168.204.136@tcp114 [14860.704176] LNet: Removed LNI 192.168.204.136@tcp115 [14861.728265] LNet: Removed LNI 192.168.204.136@tcp116 [14862.752695] LNet: Removed LNI 192.168.204.136@tcp117 [14863.776233] LNet: Removed LNI 192.168.204.136@tcp118 [14864.800257] LNet: Removed LNI 192.168.204.136@tcp119 [14864.821971] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14864.829875] LNetError: 468413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14864.836948] LNet: Removed LNI 192.168.204.136@tcp120 [14865.888305] LNet: Removed LNI 192.168.204.136@tcp121 [14866.912560] LNet: Removed LNI 192.168.204.136@tcp122 [14867.936628] LNet: Removed LNI 192.168.204.136@tcp123 [14868.960219] LNet: Removed LNI 192.168.204.136@tcp124 [14869.985782] LNet: Removed LNI 192.168.204.136@tcp125 [14871.008171] LNet: Removed LNI 192.168.204.136@tcp126 [14871.012151] LNet: Removed LNI 192.168.204.136@tcp127 [14872.032412] LNet: Removed LNI 192.168.204.136@tcp128 [14873.056476] LNet: Removed LNI 192.168.204.136@tcp129 [14873.060144] LNet: Removed LNI 192.168.204.136@tcp130 [14873.070103] LNet: Removed LNI 192.168.204.136@tcp131 [14873.080389] LNet: Removed LNI 192.168.204.136@tcp132 [14873.085140] LNet: Removed LNI 192.168.204.136@tcp133 [14873.087063] LNet: Removed LNI 192.168.204.136@tcp134 [14873.109745] LNet: Removed LNI 192.168.204.136@tcp135 [14874.001253] Key type .llcrypt unregistered [14874.004538] Key type ._llcrypt unregistered [14899.554259] Key type ._llcrypt registered [14899.558445] Key type .llcrypt registered [14899.683479] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [14916.676196] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [14917.749263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14917.881123] alg: No test for adler32 (adler32-zlib) [14918.997829] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [14919.221608] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [14920.888352] Key type lgssc registered [14922.088879] Lustre: Echo OBD driver; http://www.lustre.org/ [14960.267673] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:46:07 (1788320767) [14980.932372] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [14984.337656] Key type lgssc unregistered [14984.705551] LNet: 472506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14984.711208] LNetError: 472506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14984.729453] LNet: Removed LNI 192.168.204.136@tcp [14985.695627] Key type .llcrypt unregistered [14985.698155] Key type ._llcrypt unregistered [15003.398519] Key type ._llcrypt registered [15003.400586] Key type .llcrypt registered [15003.502798] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [15004.769323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15004.821802] alg: No test for adler32 (adler32-zlib) [15005.902769] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [15006.165204] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [15007.832178] Key type lgssc registered [15009.091468] Lustre: Echo OBD driver; http://www.lustre.org/ [15018.383490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15018.415429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15019.706321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15019.745993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15019.896732] Lustre: lustre-MDT0000: new disk, initializing [15020.034600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15020.054910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15025.279785] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15037.572737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15037.808747] Lustre: 473719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15037.903179] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15037.907419] Lustre: Skipped 1 previous similar message [15038.050974] Lustre: lustre-MDT0001: new disk, initializing [15038.183573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15038.252680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15038.265776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15044.048364] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15054.549737] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15061.197157] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15069.361925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15069.826518] Lustre: lustre-OST0000: new disk, initializing [15069.834902] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15069.868187] Lustre: 475688:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15070.031708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15076.438702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15076.452476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15076.594552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15076.833436] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15087.135368] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15090.903712] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing lnet_if_list [15096.036720] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15096.075639] LNet: Added LNI 192.168.204.136@tcp10 [8/256/0/180] [15101.677389] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15101.722961] LNet: Added LNI 192.168.204.136@tcp11 [8/256/0/180] [15106.547233] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15106.578185] LNet: Added LNI 192.168.204.136@tcp12 [8/256/0/180] [15111.261246] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15111.295995] LNet: Added LNI 192.168.204.136@tcp13 [8/256/0/180] [15115.750800] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15115.795985] LNet: Added LNI 192.168.204.136@tcp14 [8/256/0/180] [15120.491343] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15120.513271] LNet: Added LNI 192.168.204.136@tcp15 [8/256/0/180] [15125.156542] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15125.200980] LNet: Added LNI 192.168.204.136@tcp16 [8/256/0/180] [15129.894758] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15129.918421] LNet: Added LNI 192.168.204.136@tcp17 [8/256/0/180] [15134.970837] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15135.007555] LNet: Added LNI 192.168.204.136@tcp18 [8/256/0/180] [15140.239126] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15140.261328] LNet: Added LNI 192.168.204.136@tcp19 [8/256/0/180] [15145.696773] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15145.736342] LNet: Added LNI 192.168.204.136@tcp20 [8/256/0/180] [15151.245321] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15151.269799] LNet: Added LNI 192.168.204.136@tcp21 [8/256/0/180] [15156.773105] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15156.814165] LNet: Added LNI 192.168.204.136@tcp22 [8/256/0/180] [15162.263368] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15162.313454] LNet: Added LNI 192.168.204.136@tcp23 [8/256/0/180] [15167.981326] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15168.013306] LNet: Added LNI 192.168.204.136@tcp24 [8/256/0/180] [15173.231248] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15173.256596] LNet: Added LNI 192.168.204.136@tcp25 [8/256/0/180] [15178.925885] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15178.953539] LNet: Added LNI 192.168.204.136@tcp26 [8/256/0/180] [15185.622861] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15185.659136] LNet: Added LNI 192.168.204.136@tcp27 [8/256/0/180] [15192.108231] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15192.131915] LNet: Added LNI 192.168.204.136@tcp28 [8/256/0/180] [15198.839869] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15198.859868] LNet: Added LNI 192.168.204.136@tcp29 [8/256/0/180] [15204.946848] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15204.972873] LNet: Added LNI 192.168.204.136@tcp30 [8/256/0/180] [15212.989094] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15213.044679] LNet: Added LNI 192.168.204.136@tcp31 [8/256/0/180] [15221.752994] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15221.834731] LNet: Added LNI 192.168.204.136@tcp32 [8/256/0/180] [15230.914957] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15230.971636] LNet: Added LNI 192.168.204.136@tcp33 [8/256/0/180] [15238.345951] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15238.401967] LNet: Added LNI 192.168.204.136@tcp34 [8/256/0/180] [15246.520717] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15246.545740] LNet: Added LNI 192.168.204.136@tcp35 [8/256/0/180] [15255.383698] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15255.414125] LNet: Added LNI 192.168.204.136@tcp36 [8/256/0/180] [15262.712321] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15262.734412] LNet: Added LNI 192.168.204.136@tcp37 [8/256/0/180] [15270.085336] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15270.107938] LNet: Added LNI 192.168.204.136@tcp38 [8/256/0/180] [15278.797607] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15278.849535] LNet: Added LNI 192.168.204.136@tcp39 [8/256/0/180] [15287.387763] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15287.437535] LNet: Added LNI 192.168.204.136@tcp40 [8/256/0/180] [15296.718292] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15296.764336] LNet: Added LNI 192.168.204.136@tcp41 [8/256/0/180] [15305.651612] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15305.700741] LNet: Added LNI 192.168.204.136@tcp42 [8/256/0/180] [15314.143555] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15314.172779] LNet: Added LNI 192.168.204.136@tcp43 [8/256/0/180] [15321.690522] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15321.713707] LNet: Added LNI 192.168.204.136@tcp44 [8/256/0/180] [15329.880580] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15329.927279] LNet: Added LNI 192.168.204.136@tcp45 [8/256/0/180] [15339.344484] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15339.378540] LNet: Added LNI 192.168.204.136@tcp46 [8/256/0/180] [15348.443606] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15348.484669] LNet: Added LNI 192.168.204.136@tcp47 [8/256/0/180] [15357.003475] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15357.027462] LNet: Added LNI 192.168.204.136@tcp48 [8/256/0/180] [15366.706499] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15366.765887] LNet: Added LNI 192.168.204.136@tcp49 [8/256/0/180] [15376.344252] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15376.375224] LNet: Added LNI 192.168.204.136@tcp50 [8/256/0/180] [15396.833530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15396.845873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15396.876145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15399.400884] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15399.416198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15399.448633] Lustre: Skipped 2 previous similar messages [15401.746168] Lustre: server umount lustre-MDT0000 complete [15405.872315] LustreError: 473711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788321215 with bad export cookie 9496623335225744226 [15405.882741] LustreError: 473711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15405.883183] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15406.533029] Lustre: server umount lustre-MDT0001 complete [15424.642083] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [15426.018275] Lustre: 472994:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788321219/real 1788321219] req@00000000847fd225 x1875190288133248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788321235 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15426.056612] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15426.071711] Lustre: Skipped 1 previous similar message [15427.041389] Lustre: 472993:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788321220/real 1788321220] req@00000000bb7c1455 x1875190288133376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788321236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15430.116204] Lustre: 472992:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788321224/real 1788321224] req@000000005ee0c3b5 x1875190288133504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788321240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15433.214521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15436.256435] Lustre: 472993:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788321230/real 1788321230] req@00000000cbbc01be x1875190288133888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788321246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15436.275319] Lustre: 472993:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15441.440202] Lustre: 472993:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788321235/real 1788321235] req@00000000d3996119 x1875190288134016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788321251 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15450.720830] LustreError: 472990:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009b84b2ee x1875190288134656/t0(0) o250->MGC192.168.204.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15450.967755] LustreError: 475682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15450.987321] LustreError: 475682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15451.031975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15455.074165] LustreError: 475683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15455.094483] LustreError: 475683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15456.101115] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15456.102900] LustreError: 475682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15456.104127] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15461.356089] LustreError: 475683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15466.024372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15466.530489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15470.649702] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15471.300961] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15480.858384] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15488.219870] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15511.528573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15511.532991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15511.548061] Lustre: Skipped 1 previous similar message [15511.569390] Lustre: Skipped 1 previous similar message [15516.655624] LustreError: 476016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15516.681920] LustreError: 476016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15516.944470] Lustre: server umount lustre-OST0000 complete [15521.769612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15521.800078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15521.807531] Lustre: Skipped 1 previous similar message [15526.882332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15526.899962] Lustre: Skipped 1 previous similar message [15528.214732] Lustre: server umount lustre-MDT0000 complete [15532.021206] LustreError: 500209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15532.055894] LustreError: 500209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15532.877972] LustreError: 473712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788321342 with bad export cookie 9496623335225746844 [15532.882873] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15532.899645] LustreError: 473712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15533.236112] Lustre: server umount lustre-MDT0001 complete [15545.197664] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [15548.436647] Key type lgssc unregistered [15548.864438] LNet: 502739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15548.876145] LNetError: 502739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15548.883890] LNet: Removed LNI 192.168.204.136@tcp [15548.886722] LNet: Removed LNI 192.168.204.136@tcp10 [15548.890555] LNet: Removed LNI 192.168.204.136@tcp11 [15548.894625] LNet: Removed LNI 192.168.204.136@tcp12 [15548.898309] LNet: Removed LNI 192.168.204.136@tcp13 [15548.901557] LNet: Removed LNI 192.168.204.136@tcp14 [15548.905551] LNet: Removed LNI 192.168.204.136@tcp15 [15548.912788] LNet: Removed LNI 192.168.204.136@tcp16 [15548.918822] LNet: Removed LNI 192.168.204.136@tcp17 [15548.923751] LNet: Removed LNI 192.168.204.136@tcp18 [15548.929320] LNet: Removed LNI 192.168.204.136@tcp19 [15548.937566] LNet: Removed LNI 192.168.204.136@tcp20 [15548.943830] LNet: Removed LNI 192.168.204.136@tcp21 [15548.947188] LNet: Removed LNI 192.168.204.136@tcp22 [15548.953633] LNet: Removed LNI 192.168.204.136@tcp23 [15548.959792] LNet: Removed LNI 192.168.204.136@tcp24 [15548.965265] LNet: Removed LNI 192.168.204.136@tcp25 [15548.973430] LNet: Removed LNI 192.168.204.136@tcp26 [15548.977295] LNet: Removed LNI 192.168.204.136@tcp27 [15548.981237] LNet: Removed LNI 192.168.204.136@tcp28 [15548.986855] LNet: Removed LNI 192.168.204.136@tcp29 [15548.991613] LNet: Removed LNI 192.168.204.136@tcp30 [15548.994549] LNet: Removed LNI 192.168.204.136@tcp31 [15548.998213] LNet: Removed LNI 192.168.204.136@tcp32 [15549.005483] LNet: Removed LNI 192.168.204.136@tcp33 [15549.011853] LNet: Removed LNI 192.168.204.136@tcp34 [15549.014887] LNet: Removed LNI 192.168.204.136@tcp35 [15549.020501] LNet: Removed LNI 192.168.204.136@tcp36 [15549.028846] LNet: Removed LNI 192.168.204.136@tcp37 [15549.034786] LNet: Removed LNI 192.168.204.136@tcp38 [15549.037618] LNet: Removed LNI 192.168.204.136@tcp39 [15549.040562] LNet: Removed LNI 192.168.204.136@tcp40 [15549.043388] LNet: Removed LNI 192.168.204.136@tcp41 [15549.046264] LNet: Removed LNI 192.168.204.136@tcp42 [15549.049266] LNet: Removed LNI 192.168.204.136@tcp43 [15549.052245] LNet: Removed LNI 192.168.204.136@tcp44 [15549.054562] LNet: Removed LNI 192.168.204.136@tcp45 [15549.058994] LNet: Removed LNI 192.168.204.136@tcp46 [15549.062422] LNet: Removed LNI 192.168.204.136@tcp47 [15549.067527] LNet: Removed LNI 192.168.204.136@tcp48 [15549.074233] LNet: Removed LNI 192.168.204.136@tcp49 [15549.086550] LNet: Removed LNI 192.168.204.136@tcp50 [15550.030056] Key type .llcrypt unregistered [15550.038186] Key type ._llcrypt unregistered [15572.351778] Key type ._llcrypt registered [15572.362387] Key type .llcrypt registered [15572.521153] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [15588.547428] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [15590.006457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15590.029360] alg: No test for adler32 (adler32-zlib) [15591.046955] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [15591.342849] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [15593.056203] Key type lgssc registered [15594.449912] Lustre: Echo OBD driver; http://www.lustre.org/ [15634.409861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15634.493564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15635.894264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15635.961392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15636.084334] Lustre: lustre-MDT0000: new disk, initializing [15636.252276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15636.298740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15641.237936] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15653.425384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15653.554746] Lustre: 505939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15653.612178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15653.616600] Lustre: Skipped 1 previous similar message [15653.716729] Lustre: lustre-MDT0001: new disk, initializing [15653.799816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15653.832077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15653.838391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15658.383449] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15668.875104] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15677.708364] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15688.462259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15688.673569] Lustre: *** cfs_fail_loc=90f, val=0*** [15688.799941] Lustre: lustre-OST0000: new disk, initializing [15688.804537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15688.811651] Lustre: 507955:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15688.895607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15697.469936] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15698.605474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15698.644839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15698.828191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15708.094836] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15719.395493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15719.400566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15719.403694] LustreError: Skipped 1 previous similar message [15719.410378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15719.425519] Lustre: Skipped 1 previous similar message [15719.445483] Lustre: Skipped 1 previous similar message [15722.178720] Lustre: server umount lustre-OST0000 complete [15729.641628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15729.662274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15729.670453] Lustre: Skipped 1 previous similar message [15734.756098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15734.763013] Lustre: Skipped 1 previous similar message [15739.875558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15739.886224] Lustre: Skipped 1 previous similar message [15740.384258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15740.689820] Lustre: server umount lustre-MDT0000 complete [15744.549231] LustreError: 505929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788321554 with bad export cookie 3771423672973233668 [15744.554644] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15744.561067] LustreError: 505929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15744.958946] Lustre: server umount lustre-MDT0001 complete [15755.775543] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [15758.471988] Key type lgssc unregistered [15758.780777] LNet: 509868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15758.788540] LNetError: 509868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15758.810469] LNet: Removed LNI 192.168.204.136@tcp [15759.572421] Key type .llcrypt unregistered [15759.577738] Key type ._llcrypt unregistered [15779.716483] Key type ._llcrypt registered [15779.720927] Key type .llcrypt registered [15779.904920] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [15780.867633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15780.944103] alg: No test for adler32 (adler32-zlib) [15782.163936] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [15782.512984] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [15784.240168] Key type lgssc registered [15785.759184] Lustre: Echo OBD driver; http://www.lustre.org/ [15795.037676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15795.058578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15796.548099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15801.350718] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15806.949492] LustreError: 511144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15806.971061] LustreError: 511144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15810.966215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15811.520743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15815.994536] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15823.051618] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15830.499400] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15838.038621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15838.477544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15845.635459] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15854.349406] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15865.484953] loop0: detected capacity change from 0 to 204800000 [15865.669318] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15865.680484] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15865.693964] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15872.424946] loop0: detected capacity change from 0 to 204800000 [15872.757255] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15872.774766] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15872.788038] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15878.399364] loop0: detected capacity change from 0 to 204800000 [15884.497231] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15884.685919] Lustre: fs15246-OST0000: new disk, initializing [15884.690566] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15884.702448] Lustre: 514750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15884.785044] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15891.673517] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15902.355645] loop1: detected capacity change from 0 to 204800000 [15909.225644] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15909.435189] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15909.555253] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15909.745231] Lustre: fs15246-MDT0000: new disk, initializing [15909.985447] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15910.038309] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15910.181955] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15911.265308] Lustre: 515872:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000253913e0 x1875191101702016/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788321737 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15915.064098] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [15920.432717] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15920.452237] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15920.491482] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15940.088022] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15940.099484] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15943.613140] Lustre: server umount fs15246-MDT0000 complete [15945.201731] LustreError: 512383:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15950.308632] LustreError: 511139:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15953.861287] Lustre: server umount fs15246-OST0000 complete [15969.764167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15969.766338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15969.774718] LustreError: Skipped 1 previous similar message [15969.778037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15969.783885] Lustre: Skipped 1 previous similar message [15970.787863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15970.801811] Lustre: Skipped 2 previous similar messages [15972.352453] Lustre: server umount lustre-OST0000 complete [15980.522985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15980.534646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15980.550388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15982.680088] Lustre: server umount lustre-MDT0000 complete [15986.161412] LustreError: 511871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15986.181495] LustreError: 511871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15986.527633] LustreError: 511124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788321796 with bad export cookie 10705244063554106809 [15986.538565] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15986.550723] LustreError: 511124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15986.866276] Lustre: server umount lustre-MDT0001 complete [15999.868898] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [16003.834766] Key type lgssc unregistered [16004.190674] LNet: 518976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16004.202142] LNetError: 518976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16004.221704] LNet: Removed LNI 192.168.204.136@tcp [16005.406162] Key type .llcrypt unregistered [16005.409358] Key type ._llcrypt unregistered [16024.789537] Key type ._llcrypt registered [16024.792325] Key type .llcrypt registered [16024.935540] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [16025.780125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16025.818901] alg: No test for adler32 (adler32-zlib) [16026.959000] Lustre: Lustre: Build Version: 2.17.57_44_g3acc781 [16027.148363] LNet: Added LNI 192.168.204.136@tcp [8/256/0/180] [16028.856978] Key type lgssc registered [16029.972360] Lustre: Echo OBD driver; http://www.lustre.org/ [16069.412375] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [16078.935122] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [16118.587916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16118.609598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16119.895360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16120.007853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16120.103955] Lustre: lustre-MDT0000: new disk, initializing [16120.245826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16120.265048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16124.787220] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [16135.337645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16135.486202] Lustre: 523443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16135.566432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16135.579648] Lustre: Skipped 1 previous similar message [16135.676268] Lustre: lustre-MDT0001: new disk, initializing [16135.790836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16135.842335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16135.850511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16140.285779] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [16151.107687] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16157.794886] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16164.700198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16165.003588] Lustre: lustre-OST0000: new disk, initializing [16165.012447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16165.026603] Lustre: 525415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16165.132673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16172.141082] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_default_debug -1 all [16173.660795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16173.681488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16173.832669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16184.874088] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16192.354684] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16192.639284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16198.405069] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16198.661947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16204.776127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16204.791299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16204.848599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16204.855442] Lustre: Skipped 1 previous similar message [16209.891252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16209.904014] Lustre: Skipped 1 previous similar message [16215.008583] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16215.037725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16215.054938] Lustre: Skipped 1 previous similar message [16215.383430] Lustre: server umount lustre-OST0000 complete [16220.130994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16220.145640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16220.165625] Lustre: Skipped 2 previous similar messages [16220.185126] Lustre: Skipped 1 previous similar message [16225.250645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16226.102934] Lustre: server umount lustre-MDT0000 complete [16230.755428] LustreError: 524239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788322040 with bad export cookie 2638552231219780095 [16230.770385] LustreError: MGC192.168.204.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16230.790734] LustreError: 524239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16231.178748] Lustre: server umount lustre-MDT0001 complete [16260.408454] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing set_hostid [16272.836952] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing load_modules_local [16313.071226] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16066 sec ========= 00:08:39 (1788322119) [16315.391457] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:08:42 (1788322122) === [16319.821465] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:08:47 (1788322127) === [16366.284274] Lustre: DEBUG MARKER: oleg436-server.virtnet: executing unload_modules_local [16369.381314] Key type lgssc unregistered [16369.638313] LNet: 532068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16369.645796] LNetError: 532068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16369.682853] LNet: Removed LNI 192.168.204.136@tcp [16370.686166] Key type .llcrypt unregistered [16370.690933] Key type ._llcrypt unregistered