[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 482508647 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003204] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.006013] kvm-guest: setup PV IPIs [ 0.008654] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011115] LSM: Security Framework initializing [ 0.013035] Yama: becoming mindful. [ 0.014046] SELinux: Initializing. [ 0.015088] *** VALIDATE selinux *** [ 0.023351] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028625] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030008] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031173] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032110] *** VALIDATE tmpfs *** [ 0.033559] *** VALIDATE proc *** [ 0.034261] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036255] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037142] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043203] debug: unmapping init [mem 0xffffffff88859000-0xffffffff88860fff] [ 0.046137] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047700] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048022] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052013] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054017] ... event mask: 000000070000000f [ 0.055309] rcu: Hierarchical SRCU implementation. [ 0.057549] smp: Bringing up secondary CPUs ... [ 0.058622] x86: Booting SMP configuration: [ 0.059031] .... node #0, CPUs: #1 #2 #3 [ 0.064092] smp: Brought up 1 node, 4 CPUs [ 0.066019] smpboot: Max logical packages: 1 [ 0.067013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143000] node 0 deferred pages initialised in 72ms [ 0.146102] devtmpfs: initialized [ 0.148299] x86/mm: Memory block size: 128MB [ 0.150696] gcov: version magic: 0x41383552 [ 0.154300] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158367] pinctrl core: initialized pinctrl subsystem [ 0.160207] [ 0.160770] ************************************************************* [ 0.163013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167016] ** ** [ 0.170012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172012] ** ** [ 0.175017] ** This means that this kernel is built to expose internal ** [ 0.178029] ** IOMMU data structures, which may compromise security on ** [ 0.181014] ** your system. ** [ 0.184012] ** ** [ 0.187018] ** If you see this message and you are not debugging the ** [ 0.189012] ** kernel, report this immediately to your vendor! ** [ 0.192017] ** ** [ 0.195016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.197013] ************************************************************* [ 0.200752] NET: Registered protocol family 16 [ 0.202424] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.205063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.208073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211112] cpuidle: using governor menu [ 0.212733] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.215699] PCI: Using configuration type 1 for base access [ 0.217130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.227029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231079] cryptd: max_cpu_qlen set to 1000 [ 0.235325] ACPI: Added _OSI(Module Device) [ 0.237034] ACPI: Added _OSI(Processor Device) [ 0.238022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.240015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245326] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.253571] ACPI: Interpreter enabled [ 0.255072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.256010] ACPI: Using IOAPIC for interrupt routing [ 0.257105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.261373] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.272113] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.274059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.277022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287411] acpiphp: Slot [2] registered [ 0.289161] acpiphp: Slot [5] registered [ 0.290132] acpiphp: Slot [6] registered [ 0.292114] acpiphp: Slot [7] registered [ 0.293104] acpiphp: Slot [8] registered [ 0.295076] acpiphp: Slot [9] registered [ 0.296177] acpiphp: Slot [10] registered [ 0.298109] acpiphp: Slot [3] registered [ 0.300125] acpiphp: Slot [4] registered [ 0.301070] acpiphp: Slot [11] registered [ 0.302070] acpiphp: Slot [12] registered [ 0.303059] acpiphp: Slot [13] registered [ 0.304015] acpiphp: Slot [14] registered [ 0.305133] acpiphp: Slot [15] registered [ 0.306113] acpiphp: Slot [16] registered [ 0.307132] acpiphp: Slot [17] registered [ 0.309088] acpiphp: Slot [18] registered [ 0.310128] acpiphp: Slot [19] registered [ 0.311102] acpiphp: Slot [20] registered [ 0.313092] acpiphp: Slot [21] registered [ 0.314068] acpiphp: Slot [22] registered [ 0.315069] acpiphp: Slot [23] registered [ 0.316130] acpiphp: Slot [24] registered [ 0.318152] acpiphp: Slot [25] registered [ 0.319103] acpiphp: Slot [26] registered [ 0.320143] acpiphp: Slot [27] registered [ 0.322072] acpiphp: Slot [28] registered [ 0.323111] acpiphp: Slot [29] registered [ 0.324173] acpiphp: Slot [30] registered [ 0.326162] acpiphp: Slot [31] registered [ 0.327092] PCI host bridge to bus 0000:00 [ 0.329021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.331025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.333035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.339037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.342033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.344204] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.347156] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.350303] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362032] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.367072] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.370044] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373035] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376050] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.379754] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.382840] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.390935] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.396017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.412030] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.418000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.426000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.433080] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.440018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.454029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.465996] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.472019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.484021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.502021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.514391] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.525019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.530017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.544021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.554818] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.561023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.567017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.587023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.597000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.610016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.618024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.635019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.644353] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.653025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.663018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.684018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.695463] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.698435] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.700395] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.704412] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.706254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.711116] iommu: Default domain type: Passthrough [ 0.713583] SCSI subsystem initialized [ 0.715233] ACPI: bus type USB registered [ 0.717282] usbcore: registered new interface driver usbfs [ 0.719077] usbcore: registered new interface driver hub [ 0.720000] usbcore: registered new device driver usb [ 0.720118] pps_core: LinuxPPS API ver. 1 registered [ 0.721000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.725186] PTP clock support registered [ 0.728193] EDAC MC: Ver: 3.0.0 [ 0.731151] PCI: Using ACPI for IRQ routing [ 0.733912] NetLabel: Initializing [ 0.736019] NetLabel: domain hash size = 128 [ 0.738012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.740099] NetLabel: unlabeled traffic allowed by default [ 0.743165] vgaarb: loaded [ 0.745263] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.747034] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.755000] clocksource: Switched to clocksource kvm-clock [ 0.866914] VFS: Disk quotas dquot_6.6.0 [ 0.869163] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.872933] *** VALIDATE ramfs *** [ 0.874862] *** VALIDATE hugetlbfs *** [ 0.877096] pnp: PnP ACPI init [ 0.880237] pnp: PnP ACPI: found 6 devices [ 0.898654] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.903868] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.905429] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.907281] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.908949] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.911486] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.913950] NET: Registered protocol family 2 [ 0.916491] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.921688] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.924775] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.931428] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.935422] TCP: Hash tables configured (established 65536 bind 65536) [ 0.938449] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.941386] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.943935] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.946994] NET: Registered protocol family 1 [ 0.950455] RPC: Registered named UNIX socket transport module. [ 0.952876] RPC: Registered udp transport module. [ 0.954779] RPC: Registered tcp transport module. [ 0.956399] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.958974] NET: Registered protocol family 44 [ 0.960844] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.963327] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.965644] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.968348] PCI: CLS 0 bytes, default 64 [ 0.970278] Unpacking initramfs... [ 2.385788] debug: unmapping init [mem 0xffff88a1fcc54000-0xffff88a1fffbffff] [ 2.390475] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.393132] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.395990] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.899865] Initialise system trusted keyrings [ 2.903651] Key type blacklist registered [ 2.905974] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.916282] zbud: loaded [ 2.919641] *** VALIDATE nfs *** [ 2.920857] *** VALIDATE nfs4 *** [ 2.922726] pstore: using deflate compression [ 2.927422] Platform Keyring initialized [ 3.040864] NET: Registered protocol family 38 [ 3.043071] Key type asymmetric registered [ 3.044733] Asymmetric key parser 'x509' registered [ 3.047718] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.051154] io scheduler mq-deadline registered [ 3.052832] io scheduler kyber registered [ 3.054596] io scheduler bfq registered [ 3.056827] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.060042] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.063093] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.066269] ACPI: Power Button [PWRF] [ 3.072110] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.079794] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.093418] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.100788] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.121483] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.150125] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.179922] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.187236] Non-volatile memory driver v1.3 [ 3.188941] Linux agpgart interface v0.103 [ 3.226317] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.229493] vda: detected capacity change from 0 to 74702848 [ 3.246647] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.250422] vdb: detected capacity change from 0 to 1073741824 [ 3.269397] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.273320] vdc: detected capacity change from 0 to 2621440000 [ 3.291848] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.295614] vdd: detected capacity change from 0 to 2621440000 [ 3.311777] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.314815] vde: detected capacity change from 0 to 4294967296 [ 3.330659] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.334202] vdf: detected capacity change from 0 to 4294967296 [ 3.343076] libphy: Fixed MDIO Bus: probed [ 3.354340] usbcore: registered new interface driver usbserial_generic [ 3.357516] usbserial: USB Serial support registered for generic [ 3.363458] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.369079] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.371246] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.373923] mousedev: PS/2 mouse device common for all mice [ 3.376982] rtc_cmos 00:05: RTC can wake from S4 [ 3.380609] rtc_cmos 00:05: registered as rtc0 [ 3.380851] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.382558] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.389481] intel_pstate: CPU model not supported [ 3.392978] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.393699] hid: raw HID events driver (C) Jiri Kosina [ 3.398906] usbcore: registered new interface driver usbhid [ 3.401224] usbhid: USB HID core driver [ 3.403225] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.403281] drop_monitor: Initializing network drop monitor service [ 3.409473] Initializing XFRM netlink socket [ 3.411705] NET: Registered protocol family 10 [ 3.414615] Segment Routing with IPv6 [ 3.416289] NET: Registered protocol family 17 [ 3.418801] mpls_gso: MPLS GSO support [ 3.424595] RAS: Correctable Errors collector initialized. [ 3.427133] AVX version of gcm_enc/dec engaged. [ 3.428926] AES CTR mode by8 optimization enabled [ 3.509251] sched_clock: Marking stable (3509203713, 0)->(4475295008, -966091295) [ 3.512901] registered taskstats version 1 [ 3.515063] Loading compiled-in X.509 certificates [ 3.517316] zswap: loaded using pool lzo/zbud [ 3.542252] Key type big_key registered [ 3.554897] Key type encrypted registered [ 3.556637] ima: No TPM chip found, activating TPM-bypass! [ 3.558794] ima: Allocated hash algorithm: sha1 [ 3.560329] ima: No architecture policies found [ 3.561356] evm: Initialising EVM extended attributes: [ 3.563063] evm: security.selinux [ 3.563795] evm: security.ima [ 3.564611] evm: security.capability [ 3.565735] evm: HMAC attrs: 0x1 [ 3.567545] rtc_cmos 00:05: setting system clock to 2026-09-05 00:53:23 UTC (1788569603) [ 3.573401] debug: unmapping init [mem 0xffffffff89803000-0xffffffff899fffff] [ 3.576848] debug: unmapping init [mem 0xffffffff88582000-0xffffffff88858fff] [ 3.589076] Write protecting the kernel read-only data: 28672k [ 3.592209] debug: unmapping init [mem 0xffffffff86c03000-0xffffffff86dfffff] [ 3.594582] debug: unmapping init [mem 0xffffffff87514000-0xffffffff875fffff] [ 3.629528] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.637723] systemd[1]: Detected virtualization kvm. [ 3.639753] systemd[1]: Detected architecture x86-64. [ 3.641673] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.669278] systemd[1]: No hostname configured. [ 3.670935] systemd[1]: Set hostname to . [ 3.673242] random: systemd: uninitialized urandom read (16 bytes read) [ 3.675982] systemd[1]: Initializing machine ID from random generator. [ 3.796666] random: systemd: uninitialized urandom read (16 bytes read) [ 3.800451] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.805802] random: systemd: uninitialized urandom read (16 bytes read) [ 3.809140] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.813834] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.416308] device-mapper: uevent: version 1.0.3 [ 4.418519] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.122930] virtio_net virtio0 ens2: renamed from eth0 [ 5.123227] random: fast init done [ 5.170184] scsi host0: ata_piix [ 5.190170] scsi host1: ata_piix [ 5.192503] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.194620] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.785605] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.086154] random: crng init done [ 10.087958] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.463286] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.628769] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.919217] SELinux: Disabled at runtime. [ 11.983033] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.992924] systemd[1]: Detected virtualization kvm. [ 11.994931] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.576558] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.579985] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.584556] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.589959] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.592775] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.601769] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.610723] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ 12.664944] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.058685] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.393777] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.428573] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.550092] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.564406] EDAC sbridge: Ver: 1.1.2 [ 15.252681] Key type dns_resolver registered [ 15.564904] NFS: Registering the id_resolver key type [ 15.567044] Key type id_resolver registered [ 15.568808] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg454-server login: [ 73.616085] libcfs: loading out-of-tree module taints kernel. [ 73.721216] Key type ._llcrypt registered [ 73.723819] Key type .llcrypt registered [ 73.822474] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 94.742283] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 94.770848] hrtimer: interrupt took 9900795 ns [ 96.075751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 96.106767] alg: No test for adler32 (adler32-zlib) [ 97.802120] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 98.795108] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 100.655225] Key type lgssc registered [ 102.741511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 121.505800] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 162.041308] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 175.924280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 175.960856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.261853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 177.301810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 177.419713] Lustre: lustre-MDT0000: new disk, initializing [ 177.528793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.543702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 182.119826] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.504800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.646512] Lustre: 6526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.691341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 195.696713] Lustre: Skipped 1 previous similar message [ 195.768209] Lustre: lustre-MDT0001: new disk, initializing [ 195.887644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.928056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 195.955315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 200.921312] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.048263] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 217.839886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 218.191675] Lustre: lustre-OST0000: new disk, initializing [ 218.196920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 218.204350] Lustre: 8462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 218.314284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.999437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.005514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.163823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 226.275866] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.108736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.240331] Lustre: lustre-OST0001: new disk, initializing [ 242.243238] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 242.248581] Lustre: 9535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.305564] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 249.333111] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.979499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 250.989852] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 251.128485] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 262.518297] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 267.353565] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 273.769788] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing check_logdir /tmp/testlogs/ [ 279.615702] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing yml_node [ 284.816649] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 287.335660] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 290.007661] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 291.681452] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 20:58:09 EDT 2026 [ 310.041562] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 311.539452] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 328.164940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.179867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.182264] Lustre: Skipped 3 previous similar messages [ 328.206960] Lustre: Skipped 3 previous similar messages [ 333.280404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.291364] Lustre: Skipped 3 previous similar messages [ 333.882879] Lustre: server umount lustre-MDT0000 complete [ 336.917211] LustreError: 6516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569936 with bad export cookie 12934288819358508170 [ 336.920221] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 336.923852] LustreError: 6516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 338.400461] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.401987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 338.410328] Lustre: Skipped 1 previous similar message [ 338.416051] Lustre: Skipped 1 previous similar message [ 341.411330] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 341.425484] Lustre: Skipped 1 previous similar message [ 346.530309] LustreError: 9542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.531225] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.543348] LustreError: 9542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 351.199195] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 351.485134] Lustre: server umount lustre-MDT0001 complete [ 356.885882] Lustre: server umount lustre-OST0000 complete [ 362.591952] Lustre: server umount lustre-OST0001 complete [ 371.565749] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 383.190665] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 422.107670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.311883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 422.361985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 422.482966] Lustre: lustre-MDT0000: new disk, initializing [ 422.569252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.593171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 428.032239] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 440.228177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.392357] Lustre: 15097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 440.451340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 440.455316] Lustre: Skipped 1 previous similar message [ 440.522907] Lustre: lustre-MDT0001: new disk, initializing [ 440.639706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 440.683557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 440.696408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 445.468419] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 456.285410] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 462.981348] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 471.414696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 472.061401] Lustre: lustre-OST0000: new disk, initializing [ 472.068485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 472.080773] Lustre: 17079:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 472.264629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.811476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 473.846230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 474.182293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 481.261302] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 492.291425] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 497.934623] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 498.275540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 504.390871] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 504.629518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 509.946955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.948691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 509.950717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 509.961654] Lustre: Skipped 1 previous similar message [ 509.990865] Lustre: Skipped 2 previous similar messages [ 513.405650] Lustre: server umount lustre-OST0000 complete [ 520.163202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.176973] Lustre: Skipped 1 previous similar message [ 524.511582] Lustre: server umount lustre-MDT0000 complete [ 525.286512] LustreError: 15106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.308285] LustreError: 15106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 529.386650] LustreError: 15103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.411227] LustreError: 15103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 529.480483] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570129 with bad export cookie 12934288819358509696 [ 529.489407] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.502443] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 529.916371] Lustre: server umount lustre-MDT0001 complete [ 535.386562] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:02:13 (1788570133) [ 539.305949] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 541.904995] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:02:19 (1788570139) [ 551.967443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.818154] LustreError: 19490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.945635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 558.051888] LustreError: 19491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.846721] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 567.283300] LustreError: 19491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.309277] LustreError: 19491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 569.507783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.635475] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 584.494683] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 590.673805] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 597.681301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 597.997163] LustreError: 21311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.305116] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 613.107986] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 625.126927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 625.134068] LustreError: Skipped 1 previous similar message [ 625.140296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.150472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 625.157882] Lustre: Skipped 2 previous similar messages [ 630.372927] Lustre: server umount lustre-OST0000 complete [ 636.897615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 636.910984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.919128] Lustre: Skipped 1 previous similar message [ 649.183143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 649.440192] Lustre: server umount lustre-MDT0000 complete [ 653.608342] LustreError: 19470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570253 with bad export cookie 12934288819358510585 [ 653.626216] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.627136] LustreError: 19470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 654.038150] Lustre: server umount lustre-MDT0001 complete [ 666.506243] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 670.042514] Key type lgssc unregistered [ 670.472809] LNet: 23028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.484516] LNetError: 23028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 670.501781] LNet: Removed LNI 192.168.204.154@tcp [ 671.727313] Key type .llcrypt unregistered [ 671.730175] Key type ._llcrypt unregistered [ 698.226316] Key type ._llcrypt registered [ 698.228566] Key type .llcrypt registered [ 698.377263] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 699.811114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 699.876089] alg: No test for adler32 (adler32-zlib) [ 700.970101] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 701.334556] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 703.039692] Key type lgssc registered [ 704.274710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 713.426787] loop: module loaded [ 713.479699] loop0: detected capacity change from 0 to 204800000 [ 713.697879] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 713.721239] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 714.192792] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 720.450821] loop0: detected capacity change from 0 to 204800000 [ 720.716635] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 720.738647] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 720.760345] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 726.598290] loop0: detected capacity change from 0 to 204800000 [ 733.128643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 733.161119] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 734.423852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 734.546376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 734.698351] Lustre: lustre-MDT0000: new disk, initializing [ 734.898963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 734.954385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 742.276966] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 753.033930] loop1: detected capacity change from 0 to 204800000 [ 759.326567] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 759.746677] Lustre: lustre-OST07c6: new disk, initializing [ 759.759844] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 759.766871] Lustre: Skipped 1 previous similar message [ 759.778943] Lustre: 26218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 759.898657] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 767.110747] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 768.148684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 768.160900] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 768.204528] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 772.447668] Lustre: 25204:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 772.460835] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 772.468696] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 772.475713] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 772.486333] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 772.492783] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 788.497301] Lustre: 26226:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 788.525708] Lustre: 26226:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 788.530471] Lustre: 26226:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 788.549276] Lustre: 26226:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 788.564755] Lustre: 26226:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 788.573190] Lustre: 26226:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.601427] Lustre: 25203:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 793.615223] Lustre: 25203:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 793.631327] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 793.649408] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.657555] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 793.666304] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.672971] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 793.683599] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.694681] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 793.704031] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.714375] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.721860] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 796.964337] Lustre: 25205:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 796.986814] Lustre: 25205:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 797.000477] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 797.016455] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.025300] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 797.033519] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.050148] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 797.066822] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.092618] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 797.107039] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.117658] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 797.124664] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 802.466065] Lustre: 25205:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 802.477585] Lustre: 25205:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 802.487173] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 802.508299] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 802.517793] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 802.535386] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 802.541438] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 802.552027] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 802.564461] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 802.579109] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 802.591652] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 802.607059] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.260308] Lustre: 25203:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 811.272736] Lustre: 25203:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 811.292521] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 811.302506] Lustre: 25203:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.317884] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 811.332857] Lustre: 25203:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.338899] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 811.349555] Lustre: 25203:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.354499] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 811.363853] Lustre: 25203:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.372589] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 811.380993] Lustre: 25203:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 865.254073] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.269269] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 865.617483] Lustre: server umount lustre-OST07c6 complete [ 877.380596] Lustre: server umount lustre-MDT0000 complete [ 897.115631] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 900.428253] Key type lgssc unregistered [ 900.773901] LNet: 28776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.788690] LNetError: 28776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.805192] LNet: Removed LNI 192.168.204.154@tcp [ 901.583125] Key type .llcrypt unregistered [ 901.586460] Key type ._llcrypt unregistered [ 922.189568] Key type ._llcrypt registered [ 922.194595] Key type .llcrypt registered [ 922.321346] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 923.894597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 924.017548] alg: No test for adler32 (adler32-zlib) [ 925.340939] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 925.652057] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 927.368619] Key type lgssc registered [ 928.886944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 937.651437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 937.674108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.224191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.433641] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 949.734338] LustreError: 30047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.755965] LustreError: 30047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 954.148268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.597216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 960.381717] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 969.969522] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 977.707329] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 985.716698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 986.090273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.541981] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1002.099273] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1009.704964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1029.909146] Lustre: server umount lustre-OST0000 complete [ 1035.233728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.240518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1035.249286] Lustre: Skipped 1 previous similar message [ 1035.266262] Lustre: Skipped 1 previous similar message [ 1040.353723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1040.365785] Lustre: Skipped 1 previous similar message [ 1044.518611] Lustre: server umount lustre-MDT0000 complete [ 1045.486588] LustreError: 30048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.512952] LustreError: 30048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1046.502669] Lustre: 29358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788570630/real 1788570630] req@000000007e0ae46b x1875452126856576/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788570646 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1046.554389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.605143] LustreError: 30775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1046.631383] LustreError: 30775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1048.918183] LustreError: 30030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570648 with bad export cookie 2062356307175055276 [ 1048.922206] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1048.935937] LustreError: 30030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1049.344359] Lustre: server umount lustre-MDT0001 complete [ 1061.881245] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1065.829716] Key type lgssc unregistered [ 1066.170462] LNet: 33810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.186489] LNetError: 33810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.210355] LNet: Removed LNI 192.168.204.154@tcp [ 1067.089550] Key type .llcrypt unregistered [ 1067.091762] Key type ._llcrypt unregistered [ 1087.607679] Key type ._llcrypt registered [ 1087.612295] Key type .llcrypt registered [ 1087.714425] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1088.532252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1088.560714] alg: No test for adler32 (adler32-zlib) [ 1089.690433] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 1089.978810] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1091.671537] Key type lgssc registered [ 1093.325418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1102.607650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1102.648822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1104.117703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.362630] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1114.601478] LustreError: 35086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.632672] LustreError: 35086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1119.143608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.687061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1125.379639] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1134.443457] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1141.378504] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1149.652845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1149.945965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1158.951811] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1170.894170] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1174.530350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1189.869751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1189.874137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.884654] LustreError: Skipped 1 previous similar message [ 1189.885099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1189.937212] Lustre: Skipped 1 previous similar message [ 1194.969880] Lustre: server umount lustre-OST0000 complete [ 1201.633658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1201.639738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.659927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.673702] Lustre: Skipped 1 previous similar message [ 1205.219737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1205.220105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.758799] Lustre: server umount lustre-MDT0000 complete [ 1209.379122] LustreError: 35110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.392079] LustreError: 35110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1211.377365] LustreError: 35066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570811 with bad export cookie 933871542191857738 [ 1211.382112] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1211.393067] LustreError: 35066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1211.638852] Lustre: server umount lustre-MDT0001 complete [ 1225.838662] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1229.544935] Key type lgssc unregistered [ 1229.984235] LNet: 38650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1230.001223] LNetError: 38650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.020127] LNet: Removed LNI 192.168.204.154@tcp [ 1231.288737] Key type .llcrypt unregistered [ 1231.296715] Key type ._llcrypt unregistered [ 1257.372180] Key type ._llcrypt registered [ 1257.374313] Key type .llcrypt registered [ 1257.486085] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1258.413520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1258.474154] alg: No test for adler32 (adler32-zlib) [ 1259.550537] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 1259.831304] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1261.488142] Key type lgssc registered [ 1262.928074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1274.070671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1274.099655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.450049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.225928] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1285.604982] LustreError: 40070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1285.631469] LustreError: 40070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1290.720720] LustreError: 40069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1291.387928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.762796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1296.730307] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1305.657400] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1312.179585] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1319.493687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1319.938604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1320.961531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1326.856982] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1336.700442] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1344.626338] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 1346.530633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.534760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.555939] Lustre: Skipped 2 previous similar messages [ 1346.564708] Lustre: Skipped 2 previous similar messages [ 1349.708178] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 1350.708709] Lustre: server umount lustre-MDT0000 complete [ 1351.652911] LustreError: 40070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.674338] LustreError: 40070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1354.566374] LustreError: 40049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570954 with bad export cookie 1627717927538380119 [ 1354.571144] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1354.577499] LustreError: 40049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1354.829452] LustreError: 40065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.866825] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 1356.772813] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.777457] LustreError: 40065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.777937] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1359.795773] LustreError: 40065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1359.820668] Lustre: Skipped 1 previous similar message [ 1361.055193] Lustre: server umount lustre-MDT0001 complete [ 1382.254502] Lustre: server umount lustre-OST0000 complete [ 1401.859025] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1404.991707] Key type lgssc unregistered [ 1405.341789] LNet: 43943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1405.356220] LNetError: 43943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1405.377072] LNet: Removed LNI 192.168.204.154@tcp [ 1406.501586] Key type .llcrypt unregistered [ 1406.505186] Key type ._llcrypt unregistered [ 1434.531307] Key type ._llcrypt registered [ 1434.536497] Key type .llcrypt registered [ 1434.666250] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1435.452255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1435.522064] alg: No test for adler32 (adler32-zlib) [ 1436.592966] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 1436.792204] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1438.464452] Key type lgssc registered [ 1439.776339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1447.663414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1447.699065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.200508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.663115] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1459.687259] LustreError: 45363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1459.710761] LustreError: 45363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1461.887185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1462.362634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1467.625222] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1475.566545] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1482.580375] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1489.420594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1489.757282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.988901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1496.902835] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1505.906371] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1514.089031] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 1515.490641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1515.498960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.514763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1517.542098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1517.547281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1517.559186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.665751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1522.674293] Lustre: Skipped 2 previous similar messages [ 1527.263440] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1527.398904] Lustre: server umount lustre-OST0000 complete [ 1547.507513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.512908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.520713] Lustre: Skipped 1 previous similar message [ 1547.531997] Lustre: Skipped 3 previous similar messages [ 1551.870370] Lustre: server umount lustre-MDT0000 complete [ 1552.866391] LustreError: 46091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1552.896902] LustreError: 46091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1556.179759] LustreError: 45344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571156 with bad export cookie 15986655749257795282 [ 1556.191583] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1556.194567] LustreError: 45344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1562.679668] Lustre: server umount lustre-MDT0001 complete [ 1573.424914] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1576.010385] Key type lgssc unregistered [ 1576.319033] LNet: 49072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.326514] LNetError: 49072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.341129] LNet: Removed LNI 192.168.204.154@tcp [ 1577.200856] Key type .llcrypt unregistered [ 1577.204882] Key type ._llcrypt unregistered [ 1599.962121] Key type ._llcrypt registered [ 1599.964235] Key type .llcrypt registered [ 1600.146033] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1601.730643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1601.752565] alg: No test for adler32 (adler32-zlib) [ 1602.917185] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 1603.214969] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1604.984471] Key type lgssc registered [ 1606.465546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1615.449576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1615.460732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1617.051544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1621.904244] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1626.603389] LustreError: 50497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1626.638671] LustreError: 50497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1631.718381] LustreError: 50498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1631.778603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.069021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1636.806985] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1646.085416] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1652.548816] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1659.635495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1660.168680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1661.180815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1667.814501] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1676.479262] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1682.849139] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1686.757171] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1698.704826] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0421h21m36s [ 1700.681395] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1703.906426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1703.930467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.964465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1705.060150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1705.066025] Lustre: Skipped 4 previous similar messages [ 1709.555390] Lustre: server umount lustre-MDT0000 complete [ 1710.159062] LustreError: 50492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1713.408161] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1716.304572] LustreError: 50494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.328230] LustreError: 50494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1721.284791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.516457] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1721.877606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1721.967565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1722.926129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1727.769308] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1731.422274] Lustre: DEBUG MARKER: Wait for df (36728) ... [ 1733.463378] Lustre: DEBUG MARKER: done [ 1738.222372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.231794] Lustre: Skipped 2 previous similar messages [ 1738.238047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1738.244537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1738.246696] Lustre: Skipped 2 previous similar messages [ 1743.330393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1743.348493] Lustre: Skipped 2 previous similar messages [ 1744.193972] Lustre: server umount lustre-OST0000 complete [ 1748.448789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.449402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1748.454863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1748.466935] Lustre: Skipped 2 previous similar messages [ 1754.217828] Lustre: server umount lustre-MDT0000 complete [ 1754.529217] LustreError: 50497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1754.549751] LustreError: 50497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1758.154144] LustreError: 50479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571358 with bad export cookie 2465270861648781735 [ 1758.161427] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1758.167916] LustreError: 50479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1758.470710] Lustre: server umount lustre-MDT0001 complete [ 1769.869391] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1772.560555] Key type lgssc unregistered [ 1772.909650] LNet: 55428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.914990] LNetError: 55428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1773.935075] LNet: Removed LNI 192.168.204.154@tcp [ 1774.873824] Key type .llcrypt unregistered [ 1774.876952] Key type ._llcrypt unregistered [ 1817.403956] Key type ._llcrypt registered [ 1817.406849] Key type .llcrypt registered [ 1817.505193] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1819.331850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1819.403196] alg: No test for adler32 (adler32-zlib) [ 1820.509217] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 1820.864531] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1822.623379] Key type lgssc registered [ 1823.997954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1833.673211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1833.707943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.004671] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1835.021896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1835.207670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1840.091281] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1848.430891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1848.584796] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1848.602885] Lustre: 57637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1848.834369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1854.680897] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1868.113539] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1875.344428] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1883.223534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1883.447867] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1883.690318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.486547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1890.696463] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1898.807355] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1906.400268] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0421h25m04s [ 1908.651625] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1916.554681] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1919.178297] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1919.233723] LustreError: 57646:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1939.567783] Lustre: lustre-MDT0000: Client 5e561bad-bb30-441d-8faa-066ec49a42f4 (at 192.168.204.54@tcp) reconnecting [ 1959.247341] LustreError: 57646:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1962.159517] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1965.018813] Lustre: DEBUG MARKER: done [ 1977.315426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1977.334207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.356827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1977.845576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1977.850145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.869476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1980.030662] Lustre: server umount lustre-OST0000 complete [ 1987.059295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1987.079152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.116200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1991.690928] Lustre: server umount lustre-MDT0000 complete [ 1997.074523] LustreError: 60402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571597 with bad export cookie 16394267345275503454 [ 1997.079571] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1997.090902] LustreError: 60402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1998.149356] Lustre: server umount lustre-MDT0001 complete [ 2011.185693] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2014.784878] Key type lgssc unregistered [ 2015.104474] LNet: 61975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.123441] LNetError: 61975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.148314] LNet: Removed LNI 192.168.204.154@tcp [ 2016.054972] Key type .llcrypt unregistered [ 2016.056834] Key type ._llcrypt unregistered [ 2069.993744] Key type ._llcrypt registered [ 2069.997821] Key type .llcrypt registered [ 2070.157909] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2071.916269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2071.968371] alg: No test for adler32 (adler32-zlib) [ 2073.203575] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 2073.605898] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2075.303719] Key type lgssc registered [ 2077.093527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2081.008981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2081.031616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.301337] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2082.331946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2082.617401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.827116] Lustre: Failing over lustre-MDT0000 [ 2084.211312] Lustre: server umount lustre-MDT0000 complete [ 2092.146838] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:28:10 (1788571690) [ 2099.569960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2099.895854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.211286] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2112.993583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2113.134952] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2113.160118] Lustre: 64776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2113.409636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2117.733234] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2125.242720] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2131.240496] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2138.734128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2139.039181] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2139.293506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.871784] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2146.499409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2154.402028] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2160.473640] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2169.824812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2169.837641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.853655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2172.391198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2172.393302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2172.405829] Lustre: Skipped 1 previous similar message [ 2172.413391] Lustre: Skipped 1 previous similar message [ 2174.371329] Lustre: server umount lustre-MDT0000 complete [ 2178.388727] LustreError: 64769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571778 with bad export cookie 16795543304558584628 [ 2178.394487] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2178.411065] LustreError: 64769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2178.895461] Lustre: server umount lustre-MDT0001 complete [ 2183.033659] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2188.205364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2197.629365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.008697] Lustre: 63698:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788571782/real 1788571782] req@0000000013fa509e x1875453330629632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788571798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2199.034609] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.031249] Lustre: 63697:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788571783/real 1788571783] req@0000000057e092d6 x1875453330629760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788571799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2203.103153] Lustre: 63700:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788571787/real 1788571787] req@000000002eaf2ee5 x1875453330629888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788571803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2203.167890] LustreError: 63696:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003f724959 x1875453330630784/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2203.584705] LustreError: 66624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2203.622179] LustreError: 66624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2203.721675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.857965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2204.769316] LustreError: 66625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.779344] LustreError: 66624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.327648] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2209.824697] Lustre: 63697:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788571793/real 1788571793] req@0000000089a78331 x1875453330630272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788571809 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2209.852536] Lustre: 63697:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2213.870369] LustreError: 66625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2213.895582] LustreError: 66625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2214.880692] Lustre: 63697:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788571798/real 1788571798] req@000000007397d7bd x1875453330630400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788571814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2218.665513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.971699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2223.822751] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2224.119202] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2227.242294] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2232.666428] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2239.334658] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2247.652208] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.654525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.676642] Lustre: Skipped 1 previous similar message [ 2247.696761] Lustre: Skipped 2 previous similar messages [ 2252.774956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.783797] Lustre: Skipped 2 previous similar messages [ 2253.227161] Lustre: server umount lustre-MDT0000 complete [ 2257.158065] LustreError: 64768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571857 with bad export cookie 16795543304558586903 [ 2257.158946] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2257.166918] LustreError: 64768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2257.637371] Lustre: server umount lustre-MDT0001 complete [ 2264.923499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2273.250933] Lustre: 63697:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788571857/real 1788571857] req@00000000d21ebb0a x1875453330658304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788571873 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2273.287684] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.300863] Lustre: Skipped 1 previous similar message [ 2275.062630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.902344] LustreError: 69282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.976150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.050355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2287.766577] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2294.250571] LustreError: 71230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.273488] LustreError: 71230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2297.467271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.952315] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2303.209629] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2311.766972] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2318.982801] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2328.544405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2328.555597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.566459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2331.821454] Lustre: server umount lustre-MDT0000 complete [ 2334.690074] LustreError: 71571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.314720] LustreError: 64768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571935 with bad export cookie 16795543304558588282 [ 2335.321713] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2335.329808] LustreError: 64768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2335.681409] Lustre: server umount lustre-MDT0001 complete [ 2342.215857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2345.877879] Lustre: DEBUG MARKER: files compared the same [ 2358.828697] Lustre: server umount lustre-OST0000 complete [ 2378.165167] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2381.215420] Key type lgssc unregistered [ 2381.579728] LNet: 74511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2381.593933] LNetError: 74511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2381.636316] LNet: Removed LNI 192.168.204.154@tcp [ 2382.660507] Key type .llcrypt unregistered [ 2382.662716] Key type ._llcrypt unregistered [ 2408.418140] Key type ._llcrypt registered [ 2408.424619] Key type .llcrypt registered [ 2408.579591] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2409.821435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2409.972647] alg: No test for adler32 (adler32-zlib) [ 2411.190810] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 2411.485518] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2413.207204] Key type lgssc registered [ 2414.583359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2423.891117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2423.909353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.422110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.449111] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug +malloc all [ 2435.555704] LustreError: 75928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.578266] LustreError: 75928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2440.674117] LustreError: 75929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.524792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.019812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2447.380792] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug +malloc all [ 2456.086473] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2462.740528] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2470.133461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2470.497491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.542981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2477.360456] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug +malloc all [ 2486.179164] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2497.505692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2497.506982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.519503] LustreError: Skipped 1 previous similar message [ 2497.550445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2501.415613] Lustre: server umount lustre-OST0000 complete [ 2507.745092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.746413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2507.769788] Lustre: Skipped 2 previous similar messages [ 2507.785245] Lustre: Skipped 2 previous similar messages [ 2511.547307] Lustre: server umount lustre-MDT0000 complete [ 2512.863830] LustreError: 75925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.882480] LustreError: 75925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2515.294374] LustreError: 75909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572115 with bad export cookie 6779140311304984699 [ 2515.318742] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2515.319741] LustreError: 75909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2515.620410] Lustre: server umount lustre-MDT0001 complete [ 2529.071647] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2532.486559] Key type lgssc unregistered [ 2532.999873] LNet: 79467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2533.013165] LNetError: 79467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2533.039670] LNet: Removed LNI 192.168.204.154@tcp [ 2534.071506] Key type .llcrypt unregistered [ 2534.073463] Key type ._llcrypt unregistered [ 2560.911441] Key type ._llcrypt registered [ 2560.916291] Key type .llcrypt registered [ 2561.034261] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2562.282164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2562.425361] alg: No test for adler32 (adler32-zlib) [ 2563.538809] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 2563.761938] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2565.487177] Key type lgssc registered [ 2566.897623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2575.523097] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2575.539568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2591.137476] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2596.255357] LustreError: 80859:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 2622.879206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.520243] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2636.449537] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2643.875688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.011199] Lustre: *** cfs_fail_loc=706, val=0*** [ 2644.421585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.492106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2648.652645] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2649.572296] LustreError: 80884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.584204] LustreError: 80884:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2654.691211] LustreError: 80885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.988060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.413039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2661.503825] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2667.895235] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2673.407697] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2677.729445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2677.744506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2677.759516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2679.267383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.272271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2679.301555] Lustre: Skipped 1 previous similar message [ 2682.023585] Lustre: server umount lustre-OST0000 complete [ 2687.968077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2687.982644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.002237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2691.942856] Lustre: server umount lustre-MDT0000 complete [ 2694.629366] LustreError: 82738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.656894] LustreError: 82738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2695.793441] LustreError: 80867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572295 with bad export cookie 3659019305563075958 [ 2695.809558] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2695.820491] LustreError: 80867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2696.167309] Lustre: server umount lustre-MDT0001 complete [ 2707.887184] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2710.543790] Key type lgssc unregistered [ 2710.757788] LNet: 84477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2710.763607] LNetError: 84477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2710.775931] LNet: Removed LNI 192.168.204.154@tcp [ 2711.597240] Key type .llcrypt unregistered [ 2711.599940] Key type ._llcrypt unregistered [ 2736.505933] Key type ._llcrypt registered [ 2736.508332] Key type .llcrypt registered [ 2736.622580] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2737.971601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2738.011975] alg: No test for adler32 (adler32-zlib) [ 2739.059660] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 2739.293764] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2740.983894] Key type lgssc registered [ 2741.975069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2749.989211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2750.031630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.972305] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2763.334817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2763.986845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2768.569067] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2769.381477] LustreError: 86579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.392897] LustreError: 86579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2773.474173] LustreError: 86583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.494218] LustreError: 86583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2776.877691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2777.144283] LustreError: 86579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.174767] LustreError: 86579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2777.194846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.181787] LustreError: 86578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.195923] LustreError: 86578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2783.580571] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2787.300826] LustreError: 86582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.324088] LustreError: 86582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2791.116153] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2791.136326] Lustre: MGS: Client cac89dd2-9bb6-4b88-b97a-7e0dc60668e6 (at 0@lo) reconnecting [ 2791.160640] LustreError: 86593:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2791.474218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.597469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2795.745513] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2811.877175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2811.889660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2811.904197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2811.927307] Lustre: Skipped 1 previous similar message [ 2816.994839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2817.006244] Lustre: Skipped 1 previous similar message [ 2817.563794] Lustre: server umount lustre-OST0000 complete [ 2822.112876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.115768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2822.134870] Lustre: Skipped 2 previous similar messages [ 2822.148429] Lustre: Skipped 1 previous similar message [ 2827.237138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2827.247823] Lustre: Skipped 1 previous similar message [ 2827.947872] Lustre: server umount lustre-MDT0000 complete [ 2832.354318] LustreError: 88124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.380025] LustreError: 88124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2832.949696] Lustre: server umount lustre-MDT0001 complete [ 2836.977756] Lustre: server umount MGS complete [ 2849.408997] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:40:47 (1788572447) [ 2876.519415] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2887.373638] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2923.724581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.042245] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2938.171536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.398468] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2938.450433] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2938.535985] Lustre: lustre-MDT0001: new disk, initializing [ 2938.590188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2938.624411] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2938.632403] Lustre: 93773:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2938.644462] LustreError: 93773:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2943.160600] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2952.649107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2952.843793] Lustre: lustre-OST0000: new disk, initializing [ 2952.847534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2952.853865] Lustre: 94538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2952.915476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.050847] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2959.394244] Lustre: 94833:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000f7d08c44 x1875454028980480/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788572575 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2967.113297] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2972.917985] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2972.930043] Lustre: MGS: Client cf4d0752-cb08-4286-99d7-398eb5494444 (at 0@lo) reconnecting [ 2973.004197] Lustre: 95584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2973.023861] LustreError: 93759:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2973.037544] LustreError: 93759:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2973.063363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2973.142048] Lustre: lustre-MDT0000: new disk, initializing [ 2973.303284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.362346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2973.491942] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2974.434857] Lustre: 95608:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000ba49bc96 x1875454028987008/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788572590 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2978.729062] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2979.809073] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2983.534476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2983.547479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2983.684109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3011.049725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.071385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.291336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.306172] Lustre: Skipped 4 previous similar messages [ 3024.351454] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3024.682635] Lustre: server umount lustre-OST0000 complete [ 3031.520578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3031.521820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.546232] Lustre: Skipped 2 previous similar messages [ 3041.767407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.776711] Lustre: Skipped 5 previous similar messages [ 3043.295145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3043.426827] Lustre: server umount lustre-MDT0000 complete [ 3046.890056] LustreError: 95926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.937915] LustreError: 95926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3053.711702] Lustre: server umount lustre-MDT0001 complete [ 3057.482630] Lustre: server umount MGS complete [ 3067.910255] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:44:26 (1788572666) [ 3084.640550] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3087.432513] Key type lgssc unregistered [ 3087.733175] LNet: 98108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3087.752488] LNetError: 98108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3087.772808] LNet: Removed LNI 192.168.204.154@tcp [ 3088.781437] Key type .llcrypt unregistered [ 3088.785301] Key type ._llcrypt unregistered [ 3105.797257] Key type ._llcrypt registered [ 3105.799700] Key type .llcrypt registered [ 3105.919831] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3106.509808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3106.654939] alg: No test for adler32 (adler32-zlib) [ 3107.638713] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 3107.818121] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3109.463178] Key type lgssc registered [ 3110.604857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3119.110648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3119.162662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.692656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3125.681599] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3130.849833] LustreError: 99329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.885356] LustreError: 99329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3134.399183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.793622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3138.979851] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3147.529584] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3153.512902] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3160.663747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3161.271930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3165.375569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3167.583406] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3177.260697] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3190.241669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.243987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.269620] Lustre: Skipped 1 previous similar message [ 3190.277969] Lustre: Skipped 1 previous similar message [ 3195.381829] LustreError: 101154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3195.392749] LustreError: 101154:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3195.572252] Lustre: server umount lustre-OST0000 complete [ 3200.481490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3200.492375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3200.499140] Lustre: Skipped 1 previous similar message [ 3205.606553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3205.619297] Lustre: Skipped 1 previous similar message [ 3207.038518] Lustre: server umount lustre-MDT0000 complete [ 3210.729516] LustreError: 99329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3210.763783] LustreError: 99329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3210.869278] LustreError: 99311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572810 with bad export cookie 5451342041792177301 [ 3210.884948] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3210.897191] LustreError: 99311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3211.102583] Lustre: server umount lustre-MDT0001 complete [ 3225.756711] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3228.962766] Key type lgssc unregistered [ 3229.224562] LNet: 102870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3229.236817] LNetError: 102870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3230.310889] LNet: Removed LNI 192.168.204.154@tcp [ 3231.354458] Key type .llcrypt unregistered [ 3231.367621] Key type ._llcrypt unregistered [ 3249.857797] Key type ._llcrypt registered [ 3249.860675] Key type .llcrypt registered [ 3249.979325] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3251.022805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3251.051797] alg: No test for adler32 (adler32-zlib) [ 3252.127742] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 3252.377195] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3254.064251] Key type lgssc registered [ 3255.211326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3262.361629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3262.361792] LustreError: 103970:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3262.370624] Lustre: Skipped 1 previous similar message [ 3262.370650] LustreError: 103971:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3262.399249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.411719] LustreError: 103970:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3262.429366] LustreError: 103970:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3262.443589] LustreError: 103970:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3264.178139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3272.661953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.069430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3277.410350] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3282.901996] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing lsmod [ 3285.603330] LustreError: 105169:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3285.667492] LustreError: 105179:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3285.679697] LustreError: 105169:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4931 [ 3285.686623] LustreError: 105169:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3285.694116] LustreError: 105169:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3285.723420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3286.032068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3288.549933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3288.555508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.566321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3291.105624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.107692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3291.117396] Lustre: Skipped 1 previous similar message [ 3291.123887] Lustre: Skipped 2 previous similar messages [ 3294.600510] Lustre: server umount lustre-MDT0000 complete [ 3296.232511] LustreError: 104504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3296.278374] LustreError: 104504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3297.253158] LustreError: 104005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3297.290790] LustreError: 104005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3298.830718] LustreError: 105364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572898 with bad export cookie 3909684011749874652 [ 3298.835275] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3298.852329] LustreError: 105364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3299.367326] LustreError: 104441:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d69fe86e x1875454566879488/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3299.573826] Lustre: server umount lustre-MDT0001 complete [ 3314.296739] Lustre: server umount lustre-OST0000 complete [ 3325.204657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.800837] LustreError: 106242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3325.894534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.234788] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3331.043941] LustreError: 106243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3335.144475] LustreError: 106242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3340.450235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.904488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3345.836873] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3354.663679] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3360.517477] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3368.072306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3368.488654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3370.565639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3376.326199] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3385.926701] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3396.072153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.073502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3396.081498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3396.099072] Lustre: Skipped 1 previous similar message [ 3401.186633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3401.195628] Lustre: Skipped 1 previous similar message [ 3406.306160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3406.314461] Lustre: Skipped 1 previous similar message [ 3407.328823] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3407.642916] Lustre: server umount lustre-OST0000 complete [ 3412.450807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3412.454886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.547981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.553429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3416.579045] Lustre: Skipped 3 previous similar messages [ 3418.678499] Lustre: server umount lustre-MDT0000 complete [ 3421.664488] LustreError: 106238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3421.688060] LustreError: 106238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3423.421457] LustreError: 106225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573023 with bad export cookie 3909684011749875352 [ 3423.455763] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3423.846464] Lustre: server umount lustre-MDT0001 complete [ 3436.047196] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3439.091787] Key type lgssc unregistered [ 3439.380830] LNet: 109781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3439.389592] LNetError: 109781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3439.412546] LNet: Removed LNI 192.168.204.154@tcp [ 3440.580578] Key type .llcrypt unregistered [ 3440.583315] Key type ._llcrypt unregistered [ 3465.195929] Key type ._llcrypt registered [ 3465.199500] Key type .llcrypt registered [ 3465.298642] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3466.350287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3466.373614] alg: No test for adler32 (adler32-zlib) [ 3467.466587] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 3467.739551] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3469.455178] Key type lgssc registered [ 3470.714290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3479.577265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3479.592964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.188970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.648257] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3491.808082] LustreError: 111199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3491.833229] LustreError: 111199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3496.730976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3496.936585] LustreError: 111200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3497.297067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3501.901678] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3510.442816] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3517.292980] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3525.591518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3526.127353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3527.171169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3533.187671] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3542.783945] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3548.396669] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3553.178131] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3562.986992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3562.988928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3563.004380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3563.030847] Lustre: Skipped 1 previous similar message [ 3564.264378] Lustre: server umount lustre-OST0000 complete [ 3569.125035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3569.136071] LustreError: Skipped 1 previous similar message [ 3569.138032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3569.157735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3569.167247] Lustre: Skipped 1 previous similar message [ 3573.228319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3573.231817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3574.766533] Lustre: server umount lustre-MDT0000 complete [ 3578.314435] LustreError: 111935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573178 with bad export cookie 12733387922933039758 [ 3578.318070] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3578.323063] LustreError: 111935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3578.675455] Lustre: server umount lustre-MDT0001 complete [ 3592.045595] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3595.667527] Key type lgssc unregistered [ 3596.027602] LNet: 114841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3596.062817] LNetError: 114841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3597.095316] LNet: Removed LNI 192.168.204.154@tcp [ 3598.153452] Key type .llcrypt unregistered [ 3598.155840] Key type ._llcrypt unregistered [ 3618.788989] Key type ._llcrypt registered [ 3618.793401] Key type .llcrypt registered [ 3618.952562] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3620.101622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3620.210420] alg: No test for adler32 (adler32-zlib) [ 3621.456817] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 3621.832279] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3623.569443] Key type lgssc registered [ 3625.540495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3635.677612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3635.726376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3637.305092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.325260] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3647.460220] LustreError: 116067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3647.482568] LustreError: 116067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3652.584546] LustreError: 116066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3652.896982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3653.595042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3658.309807] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3667.588981] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3674.595406] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3683.900030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3684.426691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.295672] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3701.980084] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3708.068105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3724.768700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3724.786504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3724.831489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3726.312797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.313388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3726.363297] Lustre: Skipped 1 previous similar message [ 3730.748614] Lustre: server umount lustre-OST0000 complete [ 3735.526330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3735.537638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.556128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3741.409554] Lustre: server umount lustre-MDT0000 complete [ 3741.667242] LustreError: 116067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.700148] LustreError: 116067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3746.152437] LustreError: 116046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573346 with bad export cookie 17847114909229892821 [ 3746.155867] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3746.165791] LustreError: 116046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3746.472854] Lustre: server umount lustre-MDT0001 complete [ 3758.429873] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3762.060498] Key type lgssc unregistered [ 3762.475442] LNet: 119627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3762.484246] LNetError: 119627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3762.499325] LNet: Removed LNI 192.168.204.154@tcp [ 3763.319892] Key type .llcrypt unregistered [ 3763.321832] Key type ._llcrypt unregistered [ 3791.952276] Key type ._llcrypt registered [ 3791.958938] Key type .llcrypt registered [ 3792.103987] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3793.476307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3793.496156] alg: No test for adler32 (adler32-zlib) [ 3794.555779] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 3794.815051] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3796.511245] Key type lgssc registered [ 3797.767540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3805.987498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3806.012581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3807.584041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3811.884867] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3817.957137] LustreError: 121044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3817.984234] LustreError: 121044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3823.074452] LustreError: 121045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3823.299789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3823.902333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3829.562333] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3839.592270] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3846.371405] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3853.728270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3854.156479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.041645] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3871.355828] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3877.758496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3888.826683] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3892.661272] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3892.668194] Lustre: Skipped 2 previous similar messages [ 3897.033404] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3897.036672] Lustre: Skipped 2 previous similar messages [ 3898.478346] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3898.532065] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3904.550975] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.54@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3904.556104] Lustre: Skipped 2 previous similar messages [ 3906.204324] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.54@tcp 192.168.0.[2,10]@tcp [ 3914.859495] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.54/24@tcp in log lustre-MDT0000 [ 3914.868858] Lustre: Skipped 2 previous similar messages [ 3916.303990] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.54/24@tcp [ 3916.311156] Lustre: Skipped 1 previous similar message [ 3923.952775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.960484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3923.962088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3923.985730] Lustre: Skipped 1 previous similar message [ 3924.022195] LustreError: Skipped 1 previous similar message [ 3929.056324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3929.065995] Lustre: Skipped 2 previous similar messages [ 3934.178123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3934.193390] Lustre: Skipped 1 previous similar message [ 3938.271254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3938.444631] Lustre: server umount lustre-OST0000 complete [ 3941.861279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3941.870438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.883573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3944.419268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.548732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3949.560617] Lustre: Skipped 3 previous similar messages [ 3956.191170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3956.443878] Lustre: server umount lustre-MDT0000 complete [ 3959.783953] LustreError: 121038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3959.799776] LustreError: 121038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3960.601650] LustreError: 124604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573560 with bad export cookie 679554487013614182 [ 3960.616052] LustreError: 124604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3960.616776] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3960.980312] Lustre: server umount lustre-MDT0001 complete [ 3972.438638] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3975.342598] Key type lgssc unregistered [ 3975.579630] LNet: 125768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3975.584702] LNetError: 125768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3975.614783] LNet: Removed LNI 192.168.204.154@tcp [ 3976.703272] Key type .llcrypt unregistered [ 3976.712058] Key type ._llcrypt unregistered [ 4008.414247] Key type ._llcrypt registered [ 4008.415883] Key type .llcrypt registered [ 4008.521675] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4010.395294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4010.439475] alg: No test for adler32 (adler32-zlib) [ 4011.708687] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 4012.020028] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4013.703199] Key type lgssc registered [ 4015.087936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4023.237974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4023.279915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4024.706497] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4024.732460] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.54/24@tcp [ 4024.738279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4029.795433] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4035.045655] LustreError: 127335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.084172] LustreError: 127335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4039.673383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4040.081727] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4040.149182] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.54/24@tcp [ 4040.192674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4045.436472] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4053.467314] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4060.121363] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4067.661656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4068.163836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4069.191893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4074.732833] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4083.850557] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4100.070619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4100.078696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4100.078994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4100.091249] Lustre: Skipped 1 previous similar message [ 4100.131032] LustreError: Skipped 1 previous similar message [ 4101.571477] Lustre: server umount lustre-OST0000 complete [ 4105.187500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4105.202168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4105.209451] Lustre: Skipped 2 previous similar messages [ 4105.253725] Lustre: Skipped 1 previous similar message [ 4110.306152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4110.315356] Lustre: Skipped 1 previous similar message [ 4111.552667] Lustre: server umount lustre-MDT0000 complete [ 4115.427099] LustreError: 128068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4115.439284] LustreError: 128068:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4116.459655] LustreError: 127314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573716 with bad export cookie 16704752170316799870 [ 4116.467676] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4116.806450] Lustre: server umount lustre-MDT0001 complete [ 4132.187642] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4135.013199] Key type lgssc unregistered [ 4135.309118] LNet: 130937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4135.317184] LNetError: 130937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4135.336818] LNet: Removed LNI 192.168.204.154@tcp [ 4136.192153] Key type .llcrypt unregistered [ 4136.196683] Key type ._llcrypt unregistered [ 4162.076913] Key type ._llcrypt registered [ 4162.083329] Key type .llcrypt registered [ 4162.202344] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4180.226700] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4181.124126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4181.135191] alg: No test for adler32 (adler32-zlib) [ 4182.152156] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 4182.374705] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4184.199182] Key type lgssc registered [ 4185.214387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4223.394487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4223.422261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4224.668972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4224.697623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4224.765525] Lustre: lustre-MDT0000: new disk, initializing [ 4224.836663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4224.856979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4229.926682] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4240.583260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4240.684372] Lustre: 134339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4240.734165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4240.739645] Lustre: Skipped 1 previous similar message [ 4240.816484] Lustre: lustre-MDT0001: new disk, initializing [ 4240.903859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4240.940449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4240.950413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4245.521252] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4255.717757] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4261.995540] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4268.675575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4269.011404] Lustre: lustre-OST0000: new disk, initializing [ 4269.014354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4269.018389] Lustre: 136309:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4269.098515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4277.080306] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4278.940758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4278.960811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4279.111903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4286.871858] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4290.898857] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4291.116585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4296.542862] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4296.779759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4299.755989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.761201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4299.763514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4299.785268] Lustre: Skipped 1 previous similar message [ 4299.848637] Lustre: Skipped 1 previous similar message [ 4304.868676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4304.871044] Lustre: Skipped 1 previous similar message [ 4309.988648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4309.995785] Lustre: Skipped 1 previous similar message [ 4313.055102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4313.214952] Lustre: server umount lustre-OST0000 complete [ 4317.664709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4317.668711] LustreError: Skipped 1 previous similar message [ 4317.674099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4317.687844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4320.227658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4323.667944] Lustre: server umount lustre-MDT0000 complete [ 4328.751955] LustreError: 134331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573928 with bad export cookie 7291694853529550010 [ 4328.758926] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4328.776158] LustreError: 134331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4329.109897] Lustre: server umount lustre-MDT0001 complete [ 4339.303341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4339.587098] LustreError: 138528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4339.609047] LustreError: 138528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4339.651818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4344.252832] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4344.805260] LustreError: 138529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4349.926988] LustreError: 138528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.349689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.902127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4358.525096] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4366.634671] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4373.867167] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4380.952277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4381.231057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4388.060245] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4395.454963] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4399.934712] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4400.163310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4404.908271] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4405.138524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4414.158035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4414.349570] Lustre: lustre-OST0001: new disk, initializing [ 4414.354104] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4414.362630] Lustre: 141899:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4414.451708] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4421.437685] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4421.818087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4421.838537] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4421.949988] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4430.207463] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4430.422409] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4435.441909] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4435.698353] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4442.164570] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fab43237000.ost_server_uuid 50 [ 4443.812867] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fab43237000.ost_server_uuid in FULL state after 0 sec [ 4455.235210] Lustre: server umount lustre-OST0001 complete [ 4457.959409] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4457.961160] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.970805] LustreError: Skipped 1 previous similar message [ 4457.975810] LustreError: 140348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.994867] Lustre: Skipped 1 previous similar message [ 4458.060411] LustreError: 140348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4460.002137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4460.026421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4460.039144] Lustre: Skipped 2 previous similar messages [ 4463.095816] LustreError: 142280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.121340] LustreError: 142280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4465.593959] Lustre: server umount lustre-OST0000 complete [ 4471.780140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4471.791401] LustreError: Skipped 1 previous similar message [ 4471.799810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4471.816012] Lustre: Skipped 2 previous similar messages [ 4471.823958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4471.834375] Lustre: Skipped 3 previous similar messages [ 4476.326634] Lustre: server umount lustre-MDT0000 complete [ 4478.434101] LustreError: 138529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4478.497976] LustreError: 138529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4480.584757] LustreError: 138510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574080 with bad export cookie 7291694853529550899 [ 4480.592270] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4480.610330] LustreError: 138510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4480.959945] Lustre: server umount lustre-MDT0001 complete [ 4505.100814] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4508.545529] Key type lgssc unregistered [ 4508.944963] LNet: 144904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4508.953175] LNetError: 144904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4509.991831] LNet: Removed LNI 192.168.204.154@tcp [ 4511.087901] Key type .llcrypt unregistered [ 4511.100445] Key type ._llcrypt unregistered [ 4559.645052] Key type ._llcrypt registered [ 4559.648947] Key type .llcrypt registered [ 4559.774521] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4576.058688] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4577.008208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4577.054674] alg: No test for adler32 (adler32-zlib) [ 4578.019048] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 4578.300442] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4579.959209] Key type lgssc registered [ 4580.991099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4621.139295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4621.197333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.675339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4622.721810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4622.834335] Lustre: lustre-MDT0000: new disk, initializing [ 4622.939192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.962204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4627.061866] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4637.114196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4637.263875] Lustre: 149068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4637.342979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4637.347035] Lustre: Skipped 1 previous similar message [ 4637.457122] Lustre: lustre-MDT0001: new disk, initializing [ 4637.637678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4637.688506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4637.711674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4643.809969] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4655.293665] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4661.357413] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4668.718971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4669.139028] Lustre: lustre-OST0000: new disk, initializing [ 4669.143498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4669.157588] Lustre: 151036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4669.305640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4674.707613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4674.727657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4674.829617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4677.209444] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4687.374342] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4693.337766] Lustre: Failing over lustre-OST0000 [ 4693.671736] Lustre: server umount lustre-OST0000 complete [ 4710.367627] Lustre: 147289:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788574294/real 1788574294] req@00000000380469e7 x1875455957304704/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788574310 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4710.419339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4711.457707] Lustre: 147291:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788574295/real 1788574295] req@00000000ef773692 x1875455957304832/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788574311 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4711.458174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4711.496497] Lustre: 147291:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4712.899194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4713.215811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4713.244968] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4717.281316] Lustre: 147292:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788574300/real 1788574300] req@00000000122344d0 x1875455957306368/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788574316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4720.042185] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4720.358785] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4721.375320] Lustre: 147292:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788574305/real 1788574305] req@0000000057cce2c3 x1875455957307392/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788574321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4721.435560] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4721.436700] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4721.438789] Lustre: 147292:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4723.985509] Lustre: Failing over lustre-MDT0000 [ 4724.357834] Lustre: server umount lustre-MDT0000 complete [ 4724.707217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4724.712435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4727.056076] Lustre: 147291:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788574310/real 1788574310] req@000000004193c9ea x1875455957308416/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788574326 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4727.105385] Lustre: 147291:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4730.848924] LustreError: 150132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4730.886068] LustreError: 150132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4735.975017] LustreError: 149075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4736.006211] LustreError: 149075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4741.087176] Lustre: 147292:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788574325/real 1788574325] req@000000004553144d x1875455957319936/t0(0) o400->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788574341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4741.101770] LustreError: 149075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.123557] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4741.191793] LustreError: 149075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4744.590478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4746.209906] LustreError: 149079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4746.233415] LustreError: 149079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4751.329223] LustreError: 147288:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fa040b7d x1875455957324672/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4751.355232] LustreError: 149073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.365118] LustreError: 149073:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4751.430378] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (not set up) [ 4751.522996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4751.587753] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4755.663400] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4756.550852] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4757.011776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4757.027647] Lustre: Skipped 1 previous similar message [ 4757.061059] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4757.123370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4767.247728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.252501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4767.268498] Lustre: Skipped 3 previous similar messages [ 4767.277139] Lustre: Skipped 1 previous similar message [ 4771.856542] Lustre: server umount lustre-OST0000 complete [ 4777.449075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.453931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4777.455382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4777.465040] Lustre: Skipped 1 previous similar message [ 4782.560445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4782.578464] Lustre: Skipped 2 previous similar messages [ 4782.751270] Lustre: server umount lustre-MDT0000 complete [ 4786.840988] LustreError: 149059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574386 with bad export cookie 9327961348416385008 [ 4786.847615] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4786.858436] LustreError: 149059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4787.363029] Lustre: server umount lustre-MDT0001 complete [ 4799.714550] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4803.053724] Key type lgssc unregistered [ 4803.394131] LNet: 154698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4803.405356] LNetError: 154698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4803.430876] LNet: Removed LNI 192.168.204.154@tcp [ 4804.394854] Key type .llcrypt unregistered [ 4804.398716] Key type ._llcrypt unregistered [ 4848.399804] Key type ._llcrypt registered [ 4848.406411] Key type .llcrypt registered [ 4848.577880] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4849.953692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4850.067625] alg: No test for adler32 (adler32-zlib) [ 4851.296737] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 4851.435863] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4853.114590] Key type lgssc registered [ 4854.428406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4863.518153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4863.531593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.979962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4870.321900] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4874.212401] LustreError: 156223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4874.243452] LustreError: 156223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4879.329720] LustreError: 156224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4880.946478] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4884.452647] LustreError: 156223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4885.062582] LustreError: 156252:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4885.079852] LustreError: 156252:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4885.087532] LustreError: 156252:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4885.445991] Lustre: server umount lustre-MDT0000 complete [ 4902.080586] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4914.751948] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4955.679550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.920925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4955.944690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4956.028870] Lustre: lustre-MDT0000: new disk, initializing [ 4956.117550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.149112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4960.526509] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4970.009748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.192834] Lustre: 159555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4970.280089] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4970.295629] Lustre: Skipped 1 previous similar message [ 4970.352424] Lustre: lustre-MDT0001: new disk, initializing [ 4970.455245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4970.510959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4970.527126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4976.699457] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4986.017550] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4991.586664] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4998.052829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4998.319787] Lustre: lustre-OST0000: new disk, initializing [ 4998.323283] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4998.330265] Lustre: 161523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4998.422072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5005.287413] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5006.434294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5006.444436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5006.546159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5014.677848] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5019.755754] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5021.648918] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:16:59 (1788574619) [ 5328.865507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5328.873818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5328.891670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5329.383011] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5329.399438] Lustre: Skipped 1 previous similar message [ 5329.405483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5329.412433] Lustre: Skipped 2 previous similar messages [ 5334.497201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5334.523992] Lustre: Skipped 2 previous similar messages [ 5339.620456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5339.633849] Lustre: Skipped 2 previous similar messages [ 5342.175183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5342.409707] Lustre: server umount lustre-MDT0000 complete [ 5344.737276] LustreError: 159568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5344.761848] LustreError: 159568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5346.787603] LustreError: 159548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574946 with bad export cookie 14306776872622396738 [ 5346.798871] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5346.805601] LustreError: 159548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5349.855924] LustreError: 161783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5349.857206] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.897767] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5361.119130] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5361.390568] Lustre: server umount lustre-MDT0001 complete [ 5371.945266] Lustre: server umount lustre-OST0000 complete [ 5383.165675] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 5395.138164] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5437.124563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5437.463758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5437.512914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5437.647479] Lustre: lustre-MDT0000: new disk, initializing [ 5437.804884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5437.837540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5442.787818] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5454.513776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5454.626687] Lustre: 166199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5454.682528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5454.688227] Lustre: Skipped 1 previous similar message [ 5454.777666] Lustre: lustre-MDT0001: new disk, initializing [ 5454.879826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5454.921380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5454.933770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5459.791601] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5469.889392] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5475.904882] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5483.166727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5483.448780] Lustre: lustre-OST0000: new disk, initializing [ 5483.455489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5483.466943] Lustre: 168166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5483.581812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5490.789471] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5493.830844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5493.855038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5493.950665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5501.251660] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5506.791442] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5507.020451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5512.423573] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5512.674335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5519.847389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5519.860139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.877686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5519.890779] Lustre: Skipped 3 previous similar messages [ 5521.123463] Lustre: server umount lustre-OST0000 complete [ 5524.976110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.006295] Lustre: Skipped 2 previous similar messages [ 5539.295174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5539.301523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5539.335940] Lustre: Skipped 7 previous similar messages [ 5539.616748] Lustre: server umount lustre-MDT0000 complete [ 5544.066242] LustreError: 166192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575143 with bad export cookie 14306776872622555561 [ 5544.069226] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5544.087685] LustreError: 166192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5544.622701] Lustre: server umount lustre-MDT0001 complete [ 5556.845291] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:25:54 (1788575154) [ 5575.792944] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 5586.933608] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5630.014816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.328243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5630.384918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5630.551633] Lustre: lustre-MDT0000: new disk, initializing [ 5630.661140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5630.680854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5636.313564] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5647.868528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5648.038900] Lustre: 172925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5648.149095] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5648.362535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5653.989536] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5664.143195] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5672.105586] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5680.788896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5681.134243] Lustre: lustre-OST0000: new disk, initializing [ 5681.136967] Lustre: Skipped 1 previous similar message [ 5681.140873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5681.144163] Lustre: Skipped 2 previous similar messages [ 5681.148422] Lustre: 174908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5686.853946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5686.872776] Lustre: Skipped 1 previous similar message [ 5686.881414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5686.962333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5687.646651] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5697.187959] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5707.745100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5707.746604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.755713] LustreError: Skipped 2 previous similar messages [ 5707.789693] Lustre: Skipped 1 previous similar message [ 5712.876303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5712.893439] Lustre: Skipped 3 previous similar messages [ 5713.210369] Lustre: server umount lustre-OST0000 complete [ 5732.319169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5732.526942] Lustre: server umount lustre-MDT0000 complete [ 5733.345320] LustreError: 173955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5733.363458] LustreError: 173955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5737.527672] LustreError: 172918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575337 with bad export cookie 14306776872622556450 [ 5737.530910] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5737.542199] LustreError: 172918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5738.034684] Lustre: server umount lustre-MDT0001 complete [ 5750.625826] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:29:08 (1788575348) [ 5770.524549] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 5783.613193] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5824.802814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5825.089849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5825.141532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5825.242975] Lustre: lustre-MDT0000: new disk, initializing [ 5825.340429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5825.346396] Lustre: Skipped 2 previous similar messages [ 5825.362307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5829.981694] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5840.618975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5840.697966] Lustre: 179234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5841.043996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5844.835242] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5854.639367] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5861.585332] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5869.065231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5869.434286] Lustre: 181214:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5878.361099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5878.485346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5878.766835] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5889.708261] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5899.232837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5899.235615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5899.245063] LustreError: Skipped 1 previous similar message [ 5899.249355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5899.265971] Lustre: Skipped 3 previous similar messages [ 5899.287056] Lustre: Skipped 7 previous similar messages [ 5902.164791] Lustre: server umount lustre-OST0000 complete [ 5907.424118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5920.223186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5920.376463] Lustre: server umount lustre-MDT0000 complete [ 5924.620828] LustreError: 179224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575524 with bad export cookie 14306776872622557535 [ 5924.637117] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5924.638304] LustreError: 179224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5925.014774] Lustre: server umount lustre-MDT0001 complete [ 5936.339572] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5939.088663] Key type lgssc unregistered [ 5939.346208] LNet: 183072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5939.351582] LNetError: 183072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5939.359641] LNet: Removed LNI 192.168.204.154@tcp [ 5940.157488] Key type .llcrypt unregistered [ 5940.162335] Key type ._llcrypt unregistered [ 5964.692096] Key type ._llcrypt registered [ 5964.693485] Key type .llcrypt registered [ 5964.836870] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5965.634165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5965.677059] alg: No test for adler32 (adler32-zlib) [ 5966.766744] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 5967.008272] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5968.687924] Key type lgssc registered [ 5970.204934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5980.317427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5980.349367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5981.820197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5987.564398] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5991.404939] LustreError: 184490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5991.436959] LustreError: 184490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5996.515895] LustreError: 184491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5996.693463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5997.232611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6002.785436] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6010.508430] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6016.397526] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6023.131390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6023.574603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6026.690935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6030.774541] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6039.017053] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6064.608825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6064.612273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6064.632319] Lustre: Skipped 1 previous similar message [ 6069.280212] Lustre: server umount lustre-OST0000 complete [ 6073.829397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6073.839879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6073.875772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6073.886889] Lustre: Skipped 1 previous similar message [ 6074.851257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6074.885956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6074.894147] Lustre: Skipped 1 previous similar message [ 6079.853760] Lustre: server umount lustre-MDT0000 complete [ 6079.986764] LustreError: 184491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6080.019398] LustreError: 184491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6084.051788] LustreError: 184471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575683 with bad export cookie 16658723978030691723 [ 6084.055516] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6084.067609] LustreError: 184471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6084.603316] Lustre: server umount lustre-MDT0001 complete [ 6095.452471] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6098.292424] Key type lgssc unregistered [ 6098.685607] LNet: 188051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6098.702216] LNetError: 188051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6098.723655] LNet: Removed LNI 192.168.204.154@tcp [ 6099.655648] Key type .llcrypt unregistered [ 6099.658121] Key type ._llcrypt unregistered [ 6123.267812] Key type ._llcrypt registered [ 6123.269360] Key type .llcrypt registered [ 6123.319815] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6124.179759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6124.222778] alg: No test for adler32 (adler32-zlib) [ 6125.268525] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 6125.514342] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6127.199535] Key type lgssc registered [ 6128.218992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6136.156829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6136.176235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6137.771346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6143.328102] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6148.093093] LustreError: 189467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.126824] LustreError: 189467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6153.172318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6153.191058] LustreError: 189468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6153.790257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6157.766428] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6164.532832] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6170.315474] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6177.145090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6177.463417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6183.155356] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6183.597501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6190.602402] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6196.853882] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 6199.272836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6199.283513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.295132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6199.305846] Lustre: Skipped 1 previous similar message [ 6199.325076] Lustre: Skipped 1 previous similar message [ 6201.934309] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 6202.468616] Lustre: server umount lustre-OST0000 complete [ 6211.324079] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9fab585fd000.ost_server_uuid 50 [ 6213.143418] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6229.986846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6229.993494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6230.013457] Lustre: Skipped 1 previous similar message [ 6234.169690] Lustre: server umount lustre-MDT0000 complete [ 6235.109692] LustreError: 190526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.144421] LustreError: 190526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6238.199924] LustreError: 189449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575838 with bad export cookie 16786866308586805816 [ 6238.210935] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6238.218980] LustreError: 189449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6238.750241] Lustre: server umount lustre-MDT0001 complete [ 6247.905525] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:37:26 (1788575846) [ 6255.449430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6255.844535] LustreError: 193382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6255.933946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6260.856551] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6261.219198] LustreError: 193382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6269.537986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.885824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6275.285516] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6283.615541] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6289.411834] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6296.895061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6297.283613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6300.341792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6305.227869] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6313.680040] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6321.004954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6321.280732] Lustre: lustre-OST0001: new disk, initializing [ 6321.289049] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6321.294662] Lustre: 196223:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6321.413502] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6328.384429] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6330.546151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6330.557476] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6330.664120] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6339.560350] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6345.340285] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 6345.346603] Lustre: Skipped 1 previous similar message [ 6345.700104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6345.733863] LustreError: Skipped 1 previous similar message [ 6345.741575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6349.355379] Lustre: server umount lustre-OST0000 complete [ 6350.404564] LustreError: 195205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6350.419574] LustreError: 195205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6356.084038] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6356.295499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6362.360640] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6362.661735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6366.699272] LustreError: 195203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.740779] LustreError: 195203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6376.930092] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6376.937280] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6376.947424] Lustre: Skipped 1 previous similar message [ 6376.959627] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6376.966065] Lustre: Skipped 4 previous similar messages [ 6382.950652] Lustre: server umount lustre-OST0001 complete [ 6387.169388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.189704] Lustre: Skipped 1 previous similar message [ 6393.080798] Lustre: server umount lustre-MDT0000 complete [ 6398.232867] LustreError: 197995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575998 with bad export cookie 16786866308586806887 [ 6398.239303] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6398.244031] LustreError: 197995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6398.602730] Lustre: server umount lustre-MDT0001 complete [ 6426.747956] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:40:24 (1788576024) [ 6435.138510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6435.473735] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6435.564239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6435.735884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6441.563434] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6451.920129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6452.037019] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6452.070330] Lustre: 199668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6457.789733] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6466.537749] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6473.260771] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6482.815482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6483.026866] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6490.368822] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6492.354985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6498.575739] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6505.583252] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6505.647613] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6505.779350] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6505.786454] Lustre: Skipped 2 previous similar messages [ 6510.886899] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6513.272111] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6518.254547] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6522.849085] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6522.862149] LustreError: Skipped 1 previous similar message [ 6522.871507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6522.890804] Lustre: Skipped 1 previous similar message [ 6522.898583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6522.908905] Lustre: Skipped 6 previous similar messages [ 6527.309313] Lustre: server umount lustre-OST0000 complete [ 6529.000889] LustreError: 201518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6529.028990] LustreError: 201518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6544.362983] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6544.370700] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6544.373886] LustreError: Skipped 1 previous similar message [ 6544.380790] Lustre: Skipped 1 previous similar message [ 6549.370526] Lustre: server umount lustre-OST0001 complete [ 6554.607517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6554.613912] LustreError: Skipped 1 previous similar message [ 6559.714723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6559.723738] Lustre: Skipped 8 previous similar messages [ 6559.836874] Lustre: server umount lustre-MDT0000 complete [ 6563.823065] LustreError: 200414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576163 with bad export cookie 16786866308586808294 [ 6563.835765] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6563.836337] LustreError: 200414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6564.167808] Lustre: server umount lustre-MDT0001 complete [ 6591.372704] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:43:09 (1788576189) [ 6608.271138] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 6617.647790] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6649.325388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6649.812952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6649.882615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6649.997968] Lustre: lustre-MDT0000: new disk, initializing [ 6650.123307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6650.169308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6655.057873] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6667.364187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6667.531695] Lustre: 207761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6667.586909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6667.590813] Lustre: Skipped 1 previous similar message [ 6667.670291] Lustre: lustre-MDT0001: new disk, initializing [ 6667.860814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6667.885763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6672.301846] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6682.503710] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6688.705204] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6695.709387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6695.922794] Lustre: lustre-OST0000: new disk, initializing [ 6695.929375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6695.936746] Lustre: 209732:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6701.079115] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6705.283217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6705.311611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6705.459702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6711.560512] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6715.766350] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6715.982135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6721.098884] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6721.540807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6726.114807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6726.122911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6726.127946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6726.142949] Lustre: Skipped 4 previous similar messages [ 6726.159072] Lustre: Skipped 1 previous similar message [ 6729.636665] Lustre: server umount lustre-OST0000 complete [ 6740.457474] Lustre: server umount lustre-MDT0000 complete [ 6741.474035] LustreError: 207768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6741.492030] LustreError: 207768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6744.094226] LustreError: 207754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576344 with bad export cookie 16786866308586810079 [ 6744.098449] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6744.109091] LustreError: 207754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6744.415955] Lustre: server umount lustre-MDT0001 complete [ 6754.022015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.633437] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6768.474827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6773.246793] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6780.161863] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6785.481079] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6791.592123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6797.772301] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6805.044943] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6810.306496] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6810.540511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6815.487232] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6815.786875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6819.808898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6819.829291] Lustre: Skipped 3 previous similar messages [ 6832.095297] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6832.189909] Lustre: server umount lustre-OST0000 complete [ 6838.880754] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6844.254589] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6874.578603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6883.406660] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6885.156179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6885.201134] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6885.218859] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6890.756070] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6892.457772] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6900.704187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6900.714546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6900.721079] LustreError: Skipped 3 previous similar messages [ 6900.732472] Lustre: Skipped 10 previous similar messages [ 6902.874414] Lustre: server umount lustre-OST0000 complete [ 6921.183138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6921.386850] Lustre: server umount lustre-MDT0000 complete [ 6925.091411] LustreError: 211930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576525 with bad export cookie 16786866308586810961 [ 6925.104268] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6925.114404] LustreError: 211930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6925.517107] Lustre: server umount lustre-MDT0001 complete [ 6934.856916] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:48:52 (1788576532) [ 6941.892058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6942.624452] LustreError: 217915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6942.641656] LustreError: 217915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6942.728618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6942.734659] Lustre: Skipped 6 previous similar messages [ 6947.602816] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6957.661882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.399432] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6971.866917] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6979.334938] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6989.204123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6996.145550] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7004.222779] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7009.929720] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7013.437076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 7018.567680] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7018.810860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7026.286191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7026.488791] Lustre: lustre-OST0001: new disk, initializing [ 7026.492430] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7026.502808] Lustre: 221310:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7033.125290] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7034.536430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7034.550031] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7034.735695] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7044.172878] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7049.704721] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7049.904328] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7055.451548] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7055.737498] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7060.449675] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7060.451217] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.464360] LustreError: Skipped 2 previous similar messages [ 7060.471379] Lustre: Skipped 4 previous similar messages [ 7072.223147] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7072.314217] Lustre: server umount lustre-OST0001 complete [ 7079.440324] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7079.666539] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7084.527318] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7084.739299] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7114.503208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7119.863429] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7119.878982] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7119.880192] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7119.897426] Lustre: Skipped 2 previous similar messages [ 7121.859667] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7130.310284] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7132.221415] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7140.107100] Lustre: server umount lustre-OST0001 complete [ 7160.808419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7160.816146] Lustre: Skipped 16 previous similar messages [ 7163.807379] Lustre: server umount lustre-OST0000 complete [ 7182.303198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7182.534609] Lustre: server umount lustre-MDT0000 complete [ 7186.716948] LustreError: 217898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576786 with bad export cookie 16786866308586811927 [ 7186.726772] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7186.727665] LustreError: 217898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7201.248435] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7201.700847] Lustre: server umount lustre-MDT0001 complete [ 7228.925903] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:53:46 (1788576826) [ 7236.333524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7236.819722] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7236.861460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7241.643376] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7250.096676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7250.229536] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7250.264960] Lustre: 226559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7254.790110] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7262.248581] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7268.221553] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7275.867729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7276.050200] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7282.656635] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7290.446224] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7301.769269] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7301.891242] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7309.202628] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7317.530921] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7319.392816] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7323.032229] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7323.191730] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7328.372263] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7328.637287] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7333.758717] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fab51bea800.ost_server_uuid 50 [ 7335.147439] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fab51bea800.ost_server_uuid in IDLE state after 0 sec [ 7335.924495] Lustre: Permanently deactivating lustre-OST0001 [ 7335.937723] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7339.184109] Lustre: Permanently reactivating lustre-OST0001 [ 7348.194031] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7348.211729] LustreError: Skipped 4 previous similar messages [ 7348.218060] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7348.229401] Lustre: Skipped 6 previous similar messages [ 7349.243587] Lustre: server umount lustre-OST0001 complete [ 7353.326266] LustreError: 228408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7353.357386] LustreError: 228408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 37 previous similar messages [ 7359.226550] Lustre: server umount lustre-OST0000 complete [ 7369.821189] Lustre: server umount lustre-MDT0000 complete [ 7374.036624] LustreError: 226549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576973 with bad export cookie 16786866308586813215 [ 7374.042629] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7374.053881] LustreError: 226549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7374.445504] Lustre: server umount lustre-MDT0001 complete [ 7397.180180] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:56:35 (1788576995) [ 7404.935349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7405.417358] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7405.494157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7405.502884] Lustre: Skipped 5 previous similar messages [ 7409.992231] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7418.524378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7418.633042] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7418.662936] Lustre: 233253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7423.364783] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7431.201700] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7437.523562] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7445.085493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7445.266504] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7445.297402] Lustre: Permanently deactivating lustre-OST0000 [ 7445.310886] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7452.068424] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7453.762202] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7461.176594] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7468.634216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7468.698215] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7468.871422] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7468.878853] Lustre: Skipped 11 previous similar messages [ 7476.052892] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7478.378057] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7483.397752] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7485.275846] Lustre: Permanently reactivating lustre-OST0000 [ 7485.297633] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7485.303711] Lustre: Skipped 2 previous similar messages [ 7493.691262] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7493.691815] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 00000000d1706f44, cur 1788577093 deadline 1788577083 last 1788577053 [ 7493.698129] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7493.701435] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7493.739674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7513.055829] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7513.155122] Lustre: server umount lustre-OST0001 complete [ 7522.987142] Lustre: server umount lustre-OST0000 complete [ 7533.503687] Lustre: server umount lustre-MDT0000 complete [ 7537.069231] LustreError: 233247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577137 with bad export cookie 16786866308586814860 [ 7537.074275] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7537.083675] LustreError: 233247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7537.391185] Lustre: server umount lustre-MDT0001 complete [ 7550.022560] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7553.328694] Key type lgssc unregistered [ 7553.729305] LNet: 238134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7553.734463] LNetError: 238134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7553.750234] LNet: Removed LNI 192.168.204.154@tcp [ 7554.722988] Key type .llcrypt unregistered [ 7554.730556] Key type ._llcrypt unregistered [ 7576.791939] Key type ._llcrypt registered [ 7576.794679] Key type .llcrypt registered [ 7576.928498] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7578.232514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7578.275194] alg: No test for adler32 (adler32-zlib) [ 7579.431985] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 7579.671638] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 7581.343237] Key type lgssc registered [ 7582.660693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7592.399553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7592.416490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7593.959407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7598.067427] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7603.171865] LustreError: 239602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7603.190911] LustreError: 239602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7606.167746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7606.285644] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7606.331341] Lustre: 239591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7606.340544] Lustre: Permanently deactivating lustre-MDT0001 [ 7606.355283] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7606.609308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7609.406368] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7609.414438] LustreError: 239632:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7609.426898] LustreError: 239632:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7609.438620] LustreError: 239632:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7611.297200] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7619.507533] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7625.140203] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7631.924832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7632.334433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7638.344526] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7638.451851] LustreError: 241443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7638.488199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7643.624565] LustreError: 241807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7647.034481] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7648.736864] LustreError: 241443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7654.860208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7655.148294] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7660.557795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7660.566095] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7662.555645] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7672.827386] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7675.929837] Lustre: Permanently reactivating lustre-MDT0001 [ 7675.953953] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7675.960522] Lustre: Skipped 2 previous similar messages [ 7682.163657] Lustre: lustre-MDT0001: Client 0877a15f-1520-4132-86f4-921fffd8698e (at 192.168.204.54@tcp) reconnecting [ 7689.636660] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7691.967137] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7699.608938] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7699.873323] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7705.647565] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7705.881153] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7709.040551] Lustre: Permanently deactivating lustre-MDT0001 [ 7709.072471] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7709.077313] Lustre: Skipped 2 previous similar messages [ 7715.468077] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7721.959044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7721.961037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7721.983167] Lustre: Skipped 3 previous similar messages [ 7721.991487] Lustre: Skipped 3 previous similar messages [ 7723.994803] Lustre: server umount lustre-MDT0000 complete [ 7727.077520] LustreError: 244054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7727.100213] LustreError: 244054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7728.043137] LustreError: 239583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577327 with bad export cookie 9856627291315160020 [ 7728.049195] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7728.055647] LustreError: 239583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7728.373450] Lustre: server umount lustre-MDT0001 complete [ 7732.066374] Lustre: server umount lustre-OST0000 complete [ 7743.415451] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:02:21 (1788577341) [ 7754.245090] Lustre: server umount lustre-OST0001 complete [ 7760.490795] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7771.175575] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7806.550520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.739586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7806.746469] Lustre: Skipped 2 previous similar messages [ 7806.765191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7806.839314] Lustre: lustre-MDT0000: new disk, initializing [ 7806.907542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7806.925941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7811.416860] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7822.605761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7822.764273] Lustre: 247871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7822.882553] LustreError: 247882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7822.936651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7822.940373] Lustre: Skipped 1 previous similar message [ 7823.001543] Lustre: lustre-MDT0001: new disk, initializing [ 7823.076122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7823.116593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7823.128205] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7828.677304] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7839.011699] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7845.134661] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7852.512437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7852.782815] Lustre: lustre-OST0000: new disk, initializing [ 7852.787924] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7852.797965] Lustre: 249839:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7852.957333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7859.725969] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7859.797675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7859.808849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7859.941279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7870.302906] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7872.686761] LustreError: 247878:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7882.176639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7882.337931] Lustre: lustre-OST0001: new disk, initializing [ 7882.341616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7882.347316] Lustre: 250953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7882.727094] LustreError: 247878:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7882.751154] LustreError: 247879:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7888.983856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7888.992031] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7889.142352] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7889.218127] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7892.848455] LustreError: 247879:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7900.027494] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7901.891454] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7904.737348] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7904.737594] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7904.753033] LustreError: Skipped 1 previous similar message [ 7904.760853] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7904.773991] Lustre: Skipped 1 previous similar message [ 7909.866678] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7909.871606] Lustre: Skipped 2 previous similar messages [ 7910.306737] Lustre: server umount lustre-OST0001 complete [ 7914.985644] LustreError: 251480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7915.006908] LustreError: 251480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7930.343219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7930.355526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7930.360561] Lustre: Skipped 1 previous similar message [ 7933.693872] Lustre: server umount lustre-OST0000 complete [ 7940.577489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7940.589680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7940.590796] Lustre: Skipped 2 previous similar messages [ 7940.604358] Lustre: Skipped 1 previous similar message [ 7943.998570] Lustre: server umount lustre-MDT0000 complete [ 7947.284319] LustreError: 247862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577547 with bad export cookie 9856627291315162617 [ 7947.302515] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7953.779134] Lustre: server umount lustre-MDT0001 complete [ 7967.407503] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7971.123469] Key type lgssc unregistered [ 7971.411233] LNet: 253001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7971.418407] LNetError: 253001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7971.435520] LNet: Removed LNI 192.168.204.154@tcp [ 7972.389275] Key type .llcrypt unregistered [ 7972.392512] Key type ._llcrypt unregistered [ 8016.612232] Key type ._llcrypt registered [ 8016.613815] Key type .llcrypt registered [ 8016.767189] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8018.250135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8018.303909] alg: No test for adler32 (adler32-zlib) [ 8019.466720] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 8019.836067] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8021.607172] Key type lgssc registered [ 8022.818956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8031.205905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8031.224362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8032.552894] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8032.587896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8032.825724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8037.749784] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8047.446849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8047.571905] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8047.621420] Lustre: 255209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8047.952793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8052.730960] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8060.785872] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8067.539771] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8075.365676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8075.507223] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8075.706828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8083.152166] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8084.629303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8091.757303] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8145.892568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8145.901613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8145.915954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8146.410735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8146.412454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8146.426951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8146.426959] Lustre: Skipped 1 previous similar message [ 8151.063324] Lustre: server umount lustre-OST0000 complete [ 8156.125866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8170.745351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8171.403991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8172.471517] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8172.495406] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8180.066267] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8186.864523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8186.876834] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8187.675472] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8228.833051] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8228.849320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8228.864929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8232.931114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8232.935358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8232.936609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8232.966216] Lustre: Skipped 1 previous similar message [ 8234.147605] Lustre: server umount lustre-OST0000 complete [ 8238.055994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8238.079278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8238.081829] Lustre: Skipped 1 previous similar message [ 8244.094915] Lustre: server umount lustre-MDT0000 complete [ 8248.519193] LustreError: 255954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577848 with bad export cookie 611846230456185529 [ 8248.524976] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8248.528968] LustreError: 255954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8249.070307] Lustre: server umount lustre-MDT0001 complete [ 8261.666251] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8265.139579] Key type lgssc unregistered [ 8265.554606] LNet: 260639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8265.560813] LNetError: 260639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8265.587224] LNet: Removed LNI 192.168.204.154@tcp [ 8266.420332] Key type .llcrypt unregistered [ 8266.426572] Key type ._llcrypt unregistered [ 8292.603848] Key type ._llcrypt registered [ 8292.605891] Key type .llcrypt registered [ 8292.688980] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8293.708844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8293.830112] alg: No test for adler32 (adler32-zlib) [ 8294.895728] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 8295.158584] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8296.855205] Key type lgssc registered [ 8298.356645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8308.256164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8308.277564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8309.710025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8314.876274] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8319.969798] LustreError: 262056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8319.999298] LustreError: 262056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8324.300189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8324.725098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8330.008824] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8337.993965] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8343.905355] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8350.990744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8351.372499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8354.424223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8357.201884] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8364.307832] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8395.749271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8395.750419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8395.756693] LustreError: Skipped 1 previous similar message [ 8395.767715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8395.778445] Lustre: Skipped 1 previous similar message [ 8399.279410] Lustre: server umount lustre-OST0000 complete [ 8405.984668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8405.986916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8405.996802] Lustre: Skipped 2 previous similar messages [ 8406.018512] Lustre: Skipped 1 previous similar message [ 8409.421991] Lustre: server umount lustre-MDT0000 complete [ 8411.106336] LustreError: 262057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8411.141121] LustreError: 262057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8413.236518] LustreError: 262039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578013 with bad export cookie 10873842250693038189 [ 8413.241346] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8413.261878] LustreError: 262039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8413.608618] Lustre: server umount lustre-MDT0001 complete [ 8424.245910] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8427.233801] Key type lgssc unregistered [ 8427.557381] LNet: 266367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8427.575360] LNetError: 266367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8427.601091] LNet: Removed LNI 192.168.204.154@tcp [ 8428.471141] Key type .llcrypt unregistered [ 8428.473672] Key type ._llcrypt unregistered [ 8445.900326] Key type ._llcrypt registered [ 8445.902655] Key type .llcrypt registered [ 8446.072627] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8447.724581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8447.740497] alg: No test for adler32 (adler32-zlib) [ 8448.872828] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 8449.090028] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8450.735243] Key type lgssc registered [ 8452.230930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8460.144667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8460.171389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8461.850357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8466.975736] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8472.036772] LustreError: 267587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8472.059949] LustreError: 267587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8477.057926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8477.156877] LustreError: 267588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.691939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8482.839915] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8490.215616] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8496.578789] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8503.438542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8503.806686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8506.880154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8509.916825] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8516.820053] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8532.454259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8532.455183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8532.480163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8532.488607] LustreError: Skipped 1 previous similar message [ 8536.868565] Lustre: server umount lustre-OST0000 complete [ 8542.697234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8542.709947] Lustre: Skipped 1 previous similar message [ 8542.711415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8542.727128] Lustre: Skipped 2 previous similar messages [ 8547.665023] Lustre: server umount lustre-MDT0000 complete [ 8547.810378] LustreError: 267587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8547.832688] LustreError: 267587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8551.554927] LustreError: 267569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578151 with bad export cookie 13033122381612584954 [ 8551.578118] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8551.581876] LustreError: 267569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8551.997568] Lustre: server umount lustre-MDT0001 complete [ 8562.974882] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8566.129251] Key type lgssc unregistered [ 8566.375804] LNet: 271268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8566.382324] LNetError: 271268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8566.395171] LNet: Removed LNI 192.168.204.154@tcp [ 8567.309396] Key type .llcrypt unregistered [ 8567.314141] Key type ._llcrypt unregistered [ 8592.038197] Key type ._llcrypt registered [ 8592.044848] Key type .llcrypt registered [ 8592.177889] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8593.096347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8593.146089] alg: No test for adler32 (adler32-zlib) [ 8594.210363] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 8594.531530] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8596.288312] Key type lgssc registered [ 8597.257601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8604.038501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8604.058597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8605.464417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8609.472933] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8615.907584] LustreError: 272688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8615.933838] LustreError: 272688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8617.678911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8617.948677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8622.046978] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8629.991569] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8636.836910] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8643.460312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8643.924558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8650.044542] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8650.106215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8657.957852] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8686.048877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8686.059196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8686.069103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8688.100929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8688.103652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8688.104503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8688.111989] Lustre: Skipped 1 previous similar message [ 8689.912171] Lustre: server umount lustre-OST0000 complete [ 8694.752565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8694.763733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8694.777332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8698.341977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8698.344277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8700.203696] Lustre: server umount lustre-MDT0000 complete [ 8703.459907] LustreError: 272689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8703.482032] LustreError: 272689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8703.751176] LustreError: 275491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578303 with bad export cookie 489919423608332379 [ 8703.757782] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8703.766760] LustreError: 275491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8704.173351] Lustre: server umount lustre-MDT0001 complete [ 8715.431463] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8718.247326] Key type lgssc unregistered [ 8718.548730] LNet: 276996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8718.557320] LNetError: 276996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8718.601023] LNet: Removed LNI 192.168.204.154@tcp [ 8719.342702] Key type .llcrypt unregistered [ 8719.347356] Key type ._llcrypt unregistered [ 8736.862258] Key type ._llcrypt registered [ 8736.864105] Key type .llcrypt registered [ 8736.975491] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8738.251809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8738.266654] alg: No test for adler32 (adler32-zlib) [ 8739.375358] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 8739.632397] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8741.328083] Key type lgssc registered [ 8742.572359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8750.404448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8750.454110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8752.060865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8755.876051] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8761.318459] LustreError: 278236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8761.346035] LustreError: 278236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8763.448258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8763.778833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8767.385283] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8775.403190] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8780.962904] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8789.099615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8789.500295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8796.678130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8797.272849] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8805.196137] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8821.490797] Lustre: server umount lustre-OST0000 complete [ 8829.413517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8829.415039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8829.428849] Lustre: Skipped 1 previous similar message [ 8829.446432] Lustre: Skipped 1 previous similar message [ 8834.534638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8834.538282] Lustre: Skipped 1 previous similar message [ 8835.636564] Lustre: server umount lustre-MDT0000 complete [ 8839.106707] LustreError: 278202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578439 with bad export cookie 13113035819689950380 [ 8839.117407] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8839.118733] LustreError: 278202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8849.537158] Lustre: server umount lustre-MDT0001 complete [ 8860.292710] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8862.964069] Key type lgssc unregistered [ 8863.313669] LNet: 281943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8863.321709] LNetError: 281943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8863.344497] LNet: Removed LNI 192.168.204.154@tcp [ 8864.034913] Key type .llcrypt unregistered [ 8864.038524] Key type ._llcrypt unregistered [ 8874.959564] Key type ._llcrypt registered [ 8874.962213] Key type .llcrypt registered [ 8875.052373] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8893.183152] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 8907.994113] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8908.760727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8908.792220] alg: No test for adler32 (adler32-zlib) [ 8909.846633] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 8910.064666] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8911.746101] Key type lgssc registered [ 8912.529082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8944.826248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8944.872937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.147468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8946.224399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8946.321590] Lustre: lustre-MDT0000: new disk, initializing [ 8946.396135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8946.413984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8950.545305] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8959.991159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8960.139476] Lustre: 285496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8960.234949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8960.238954] Lustre: Skipped 1 previous similar message [ 8960.309862] Lustre: lustre-MDT0001: new disk, initializing [ 8960.408895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8960.447717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8960.458216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8964.144428] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8973.363439] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8979.300083] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8986.593431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8986.835441] Lustre: lustre-OST0000: new disk, initializing [ 8986.843398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8986.853660] Lustre: 287463:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8986.929665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8993.250450] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8996.441575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8996.463041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8996.567583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9003.018598] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9007.929584] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9008.180261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9013.015122] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9013.193210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9017.317951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9017.318975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9017.332602] LustreError: Skipped 1 previous similar message [ 9017.347621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9017.359846] Lustre: Skipped 1 previous similar message [ 9017.394103] Lustre: Skipped 1 previous similar message [ 9021.437884] Lustre: server umount lustre-OST0000 complete [ 9027.040969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9027.050985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9027.062274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9031.699558] Lustre: server umount lustre-MDT0000 complete [ 9035.493971] LustreError: 285488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578635 with bad export cookie 5567116435246383317 [ 9035.502324] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9035.507374] LustreError: 285488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9035.785543] Lustre: server umount lustre-MDT0001 complete [ 9046.462471] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:24:04 (1788578644) [ 9054.209320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9055.009537] LustreError: 289876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9055.035811] LustreError: 289876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9055.168825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9060.205705] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9060.322015] LustreError: 289877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9064.418148] LustreError: 289876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9069.557354] LustreError: 289877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9069.699613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9070.079093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9074.178568] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9080.976991] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9085.594585] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9091.172695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9091.482392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9097.189653] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9104.909206] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9115.108388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9115.112449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9115.119660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9115.126466] Lustre: Skipped 2 previous similar messages [ 9115.134331] Lustre: Skipped 2 previous similar messages [ 9119.409861] Lustre: server umount lustre-OST0000 complete [ 9125.344771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9125.356130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9125.368176] Lustre: Skipped 1 previous similar message [ 9129.571852] Lustre: server umount lustre-MDT0000 complete [ 9130.480029] LustreError: 290612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9132.890054] LustreError: 289858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578732 with bad export cookie 5567116435246384199 [ 9132.898128] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9132.903588] LustreError: 289858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9133.176965] Lustre: server umount lustre-MDT0001 complete [ 9141.796593] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 9144.160799] Key type lgssc unregistered [ 9144.414469] LNet: 293415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9144.427171] LNetError: 293415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9145.447077] LNet: Removed LNI 192.168.204.154@tcp [ 9146.216729] Key type .llcrypt unregistered [ 9146.219255] Key type ._llcrypt unregistered [ 9167.657190] Key type ._llcrypt registered [ 9167.659794] Key type .llcrypt registered [ 9167.764914] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 9183.541975] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9184.506095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9184.652867] alg: No test for adler32 (adler32-zlib) [ 9185.717926] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 9186.045316] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 9187.784822] Key type lgssc registered [ 9189.275163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9240.694730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9240.714822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9242.060228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9242.085883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9242.181353] Lustre: lustre-MDT0000: new disk, initializing [ 9242.253427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9242.276947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9246.071618] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9256.281636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9256.407882] Lustre: 297237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9256.468465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9256.471506] Lustre: Skipped 1 previous similar message [ 9256.543570] Lustre: lustre-MDT0001: new disk, initializing [ 9256.630754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9256.663261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9256.676977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9260.720913] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9271.579373] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9277.571949] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9284.644137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9284.889252] Lustre: lustre-OST2710: new disk, initializing [ 9284.893518] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9284.904395] Lustre: 299206:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9285.010563] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9290.745586] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9292.876078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9292.887081] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9292.963833] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9299.990746] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9309.621803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9310.033204] Lustre: lustre-OST03e8: new disk, initializing [ 9310.051195] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9310.067089] Lustre: 300269:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9310.244171] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9318.985038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9318.990951] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9319.090786] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9319.413984] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9327.997171] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9335.149262] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9335.392501] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9340.514384] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9340.852284] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9345.497955] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9345.727678] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9350.479221] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9350.659679] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9355.237105] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9355.242025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9355.256525] Lustre: Skipped 3 previous similar messages [ 9355.275215] Lustre: Skipped 3 previous similar messages [ 9357.483853] Lustre: server umount lustre-MDT0000 complete [ 9361.177396] LustreError: 297231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578961 with bad export cookie 13491311410260921384 [ 9361.181393] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9361.194159] LustreError: 297231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9361.798700] Lustre: server umount lustre-MDT0001 complete [ 9375.249307] Lustre: server umount lustre-OST2710 complete [ 9389.618330] Lustre: server umount lustre-OST03e8 complete [ 9408.411465] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 9418.342203] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9456.519584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9456.780332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9456.832187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9456.951673] Lustre: lustre-MDT0000: new disk, initializing [ 9457.044332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9457.068896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9462.474226] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9475.657530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9475.856995] Lustre: 305520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9476.006662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9476.015877] Lustre: Skipped 1 previous similar message [ 9476.142383] Lustre: lustre-MDT0001: new disk, initializing [ 9476.261530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9476.318284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9476.336847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9481.421854] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9490.636207] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9495.969100] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9502.615025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9502.870257] Lustre: lustre-OST0000: new disk, initializing [ 9502.878777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9502.894936] Lustre: 307488:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9503.059908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9509.403607] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9510.493332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9510.508100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9510.645882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9518.722444] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9523.799142] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9524.016848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9529.460751] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9529.687837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9536.481675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9536.483628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9536.497899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9537.730109] Lustre: server umount lustre-OST0000 complete [ 9543.142300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9543.149437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9543.158395] Lustre: Skipped 1 previous similar message [ 9543.164722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9543.172719] Lustre: Skipped 1 previous similar message [ 9546.723847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9546.725067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9546.765788] Lustre: Skipped 1 previous similar message [ 9548.197689] Lustre: server umount lustre-MDT0000 complete [ 9551.862526] LustreError: 305528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9551.892700] LustreError: 305528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9552.738274] LustreError: 305513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579152 with bad export cookie 13491311410260923064 [ 9552.756022] LustreError: 305513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9552.761112] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9553.222532] Lustre: server umount lustre-MDT0001 complete [ 9564.079775] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:32:41 (1788579161) [ 9566.092341] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9568.055728] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:32:46 (1788579166) [ 9572.389156] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9603.695228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9603.926899] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9603.961818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9604.170208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9608.767389] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9617.461191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9617.585258] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9617.606223] Lustre: 311346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9622.828430] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9644.151121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9644.340455] Lustre: Denying initial registration attempt from nid 192.168.204.154@tcp, specified as failover [ 9644.346054] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9644.355836] LustreError: 312799:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9644.365945] LustreError: 312799:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9644.372765] LustreError: 312799:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9644.458986] Lustre: server umount lustre-OST0000 complete [ 9644.461970] LustreError: 312799:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9653.728905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9653.737042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9653.749556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9657.980706] Lustre: server umount lustre-MDT0000 complete [ 9659.363174] LustreError: 311357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9659.379312] LustreError: 311357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9662.131811] LustreError: 312088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579262 with bad export cookie 13491311410260923946 [ 9662.151996] LustreError: 312088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9662.160724] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9662.573930] Lustre: server umount lustre-MDT0001 complete [ 9675.567992] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 9678.307075] Key type lgssc unregistered [ 9678.608537] LNet: 313925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9678.613803] LNetError: 313925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9678.640289] LNet: Removed LNI 192.168.204.154@tcp [ 9679.598991] Key type .llcrypt unregistered [ 9679.602348] Key type ._llcrypt unregistered [ 9694.022343] Key type ._llcrypt registered [ 9694.027175] Key type .llcrypt registered [ 9694.216765] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9695.753223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9695.790677] alg: No test for adler32 (adler32-zlib) [ 9697.054214] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 9697.312194] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 9698.959127] Key type lgssc registered [ 9700.373927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9748.922537] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9758.334284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9758.350352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9759.594692] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9759.626583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9759.837443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9764.259612] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9772.159711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9772.289258] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9772.316544] Lustre: 316689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9772.543305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9777.149779] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9785.159346] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9791.059726] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9810.611277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9810.929100] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9811.124815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9818.380595] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9825.981350] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9831.395812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9831.396464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9831.418785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9831.436763] Lustre: Skipped 1 previous similar message [ 9834.202378] Lustre: server umount lustre-OST0000 complete [ 9839.080302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9839.097548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9839.123347] Lustre: Skipped 1 previous similar message [ 9839.128892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9841.633662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9841.634429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9844.624811] Lustre: server umount lustre-MDT0000 complete [ 9848.148993] LustreError: 316684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579448 with bad export cookie 13852939081806544377 [ 9848.157513] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9848.160955] LustreError: 316684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9848.613402] Lustre: server umount lustre-MDT0001 complete [ 9859.823042] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 9862.507918] Key type lgssc unregistered [ 9862.771827] LNet: 320268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9862.781534] LNetError: 320268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9862.802826] LNet: Removed LNI 192.168.204.154@tcp [ 9863.813659] Key type .llcrypt unregistered [ 9863.815590] Key type ._llcrypt unregistered [ 9886.826499] Key type ._llcrypt registered [ 9886.829547] Key type .llcrypt registered [ 9886.931935] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9887.659745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9887.732415] alg: No test for adler32 (adler32-zlib) [ 9888.739336] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 9889.073308] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 9890.743165] Key type lgssc registered [ 9892.014817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9899.807632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9899.831856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9901.515793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9906.284812] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9911.779642] LustreError: 321684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9911.812398] LustreError: 321684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9914.920716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9915.286235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9919.404105] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9926.238690] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9933.184782] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9943.985243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9944.579392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9952.355168] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9960.666079] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9971.175913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9971.187592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9971.187841] Lustre: Skipped 2 previous similar messages [ 9971.201031] Lustre: Skipped 2 previous similar messages [ 9972.855048] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 9975.707922] Lustre: server umount lustre-MDT0000 complete [ 9976.292075] LustreError: 321685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9976.310676] LustreError: 321685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9977.932568] LustreError: 321679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9979.390234] LustreError: 321665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579579 with bad export cookie 2988316556572282352 [ 9979.396122] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9979.404170] LustreError: 321665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9981.408496] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9981.414554] LustreError: 321680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9981.432217] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9984.417226] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9984.423579] Lustre: Skipped 1 previous similar message [ 9985.759408] Lustre: server umount lustre-MDT0001 complete [ 9992.891861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10001.454971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10005.234630] LustreError: 323517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10005.250042] LustreError: 323517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10005.332146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10005.421814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10009.835737] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10014.563943] LustreError: 325276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10014.564390] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10014.576681] LustreError: 325276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10017.407112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10017.732986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10021.691501] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10022.893350] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10028.753459] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10030.374389] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10036.459304] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10038.503846] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10046.436709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10046.456913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10046.480860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10048.491369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10048.507291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10051.448625] Lustre: server umount lustre-OST0000 complete [10058.721401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10058.729874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10058.730161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10058.751540] Lustre: Skipped 1 previous similar message [10058.784200] Lustre: Skipped 3 previous similar messages [10062.280364] Lustre: server umount lustre-MDT0000 complete [10063.841841] LustreError: 325281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10063.874907] LustreError: 325281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10066.506873] LustreError: 321665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579666 with bad export cookie 2988316556572289506 [10066.516173] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10066.519220] LustreError: 321665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10067.017855] Lustre: server umount lustre-MDT0001 complete [10081.301021] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [10086.286426] Key type lgssc unregistered [10086.705717] LNet: 327954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10086.713771] LNetError: 327954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10087.782091] LNet: Removed LNI 192.168.204.154@tcp [10089.195583] Key type .llcrypt unregistered [10089.205578] Key type ._llcrypt unregistered [10119.199753] Key type ._llcrypt registered [10119.202460] Key type .llcrypt registered [10119.337125] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [10120.773391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10120.787172] alg: No test for adler32 (adler32-zlib) [10121.885075] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [10122.156591] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [10123.896437] Key type lgssc registered [10125.364935] Lustre: Echo OBD driver; http://www.lustre.org/ [10134.606404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10134.627124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10136.157869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10140.709663] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10146.288828] LustreError: 329376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10146.321384] LustreError: 329376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10149.739833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10150.049316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10154.869849] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10162.740207] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10169.274678] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10174.318217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10174.330130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10174.352671] Lustre: Skipped 1 previous similar message [10174.376911] Lustre: Skipped 1 previous similar message [10179.563388] LustreError: 329376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10179.584902] LustreError: 329376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10179.673302] Lustre: server umount lustre-MDT0000 complete [10183.441033] LustreError: 329358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579783 with bad export cookie 14309807752576611979 [10183.460822] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10183.667511] Lustre: server umount lustre-MDT0001 complete [10200.562929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10200.914126] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10200.947299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10201.171863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10206.603725] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10217.227505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10217.366822] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10217.400339] Lustre: 331964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10217.634815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10222.819098] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10232.345868] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10238.413517] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10249.654519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10249.878833] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10249.891716] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10249.900551] LustreError: 333846:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10249.906378] LustreError: 333846:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10249.915482] LustreError: 333846:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10250.037449] Lustre: server umount lustre-OST0000 complete [10250.039958] LustreError: 333846:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10261.801972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10262.203212] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10262.472610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10270.368413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10270.494577] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10278.355119] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10286.777516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10286.868865] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10286.953358] Lustre: lustre-OST0001: new disk, initializing [10286.957762] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10286.966390] Lustre: 335390:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10287.086695] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10293.124564] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10294.419803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10294.436720] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10294.576726] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10301.065963] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10304.998749] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10305.006085] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10305.009275] LustreError: Skipped 1 previous similar message [10305.009979] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10305.039112] Lustre: Skipped 1 previous similar message [10310.115596] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10310.128699] Lustre: Skipped 2 previous similar messages [10315.250368] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10315.257832] Lustre: Skipped 1 previous similar message [10319.327400] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10319.581118] Lustre: server umount lustre-OST0001 complete [10320.360145] LustreError: 335344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10320.393647] LustreError: 335344:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10325.483297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10325.496793] LustreError: 334313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.502160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10325.502546] Lustre: Skipped 1 previous similar message [10325.526894] LustreError: 334313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10330.550724] Lustre: server umount lustre-OST0000 complete [10335.200485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10335.209984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10335.234784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10335.250564] Lustre: Skipped 1 previous similar message [10341.583418] Lustre: server umount lustre-MDT0000 complete [10345.739384] LustreError: 331957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579945 with bad export cookie 14309807752576612532 [10345.742226] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10345.746514] LustreError: 331957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10346.025749] Lustre: server umount lustre-MDT0001 complete [10357.534256] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [10360.967971] Key type lgssc unregistered [10361.309224] LNet: 337354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10361.325665] LNetError: 337354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10361.342229] LNet: Removed LNI 192.168.204.154@tcp [10362.477125] Key type .llcrypt unregistered [10362.479535] Key type ._llcrypt unregistered [10444.285265] Key type ._llcrypt registered [10444.287082] Key type .llcrypt registered [10444.473215] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [10459.479180] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [10461.112104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10461.140789] alg: No test for adler32 (adler32-zlib) [10462.289205] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [10462.589035] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [10464.255226] Key type lgssc registered [10465.663355] Lustre: Echo OBD driver; http://www.lustre.org/ [10507.403270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10507.429719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10508.972293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10509.025468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10509.152669] Lustre: lustre-MDT0000: new disk, initializing [10509.410556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10509.431814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10515.730193] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10529.904488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10530.079935] Lustre: 342293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10530.196334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10530.207886] Lustre: Skipped 1 previous similar message [10530.322519] Lustre: lustre-MDT0001: new disk, initializing [10530.442540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10530.502851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10530.524839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10536.160056] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10547.509724] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10554.326848] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10563.218190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10563.522808] Lustre: lustre-OST0000: new disk, initializing [10563.527960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10563.537113] Lustre: 344266:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10563.639450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10568.819422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10568.828540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10569.025364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10572.294731] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10583.434212] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10589.054096] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10589.440715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10595.603703] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10595.882917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10599.905177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10599.906473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10599.914206] LustreError: Skipped 1 previous similar message [10599.921316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10599.935464] Lustre: Skipped 1 previous similar message [10599.959453] Lustre: Skipped 1 previous similar message [10605.034797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10605.051299] Lustre: Skipped 1 previous similar message [10610.149403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10610.160595] Lustre: Skipped 1 previous similar message [10612.191243] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10612.450301] Lustre: server umount lustre-OST0000 complete [10617.313861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10617.330694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10617.399791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10620.393574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10623.482442] Lustre: server umount lustre-MDT0000 complete [10627.695910] LustreError: 342286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580227 with bad export cookie 18337161109035389303 [10627.703514] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10627.708464] LustreError: 342286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10628.218479] Lustre: server umount lustre-MDT0001 complete [10638.426252] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:50:36 (1788580236) [10648.640820] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:50:46 (1788580246) [10661.862922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10662.705403] LustreError: 347023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10662.727769] LustreError: 347023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10662.824831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10668.006854] LustreError: 347024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10669.180799] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10672.113479] LustreError: 347023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10677.219745] LustreError: 347024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10681.088143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.567941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10687.803449] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10696.258929] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10702.299516] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10710.033128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10710.292301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10718.115833] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10727.067313] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10740.256110] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10742.331555] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10743.982693] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10745.572878] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10747.711591] Lustre: Failing over lustre-MDT0000 [10747.883515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10747.889387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10747.904471] LustreError: 347024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10748.019432] Lustre: server umount lustre-MDT0000 complete [10758.626663] LustreError: 347024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10758.654082] LustreError: 347024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10764.770262] Lustre: 340518:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788580348/real 1788580348] req@000000009e0c9cbf x1875462127167104/t0(0) o400->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788580364 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10764.825700] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10768.398794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10775.009918] LustreError: 349321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10775.010088] LustreError: 340514:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a1c84a2d x1875462127173248/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10775.053954] LustreError: 349321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [10775.482541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10775.615194] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10776.130425] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10780.658811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10780.723174] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10780.785898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10781.084175] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10792.989739] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10794.937638] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10799.655286] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10802.273756] Lustre: Failing over lustre-MDT0000 [10802.744710] Lustre: server umount lustre-MDT0000 complete [10804.198303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10804.212153] Lustre: Skipped 4 previous similar messages [10809.317285] LustreError: 347753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10809.348264] LustreError: 347753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [10820.575174] Lustre: 340516:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788580404/real 1788580404] req@00000000966ce6f9 x1875462127190144/t0(0) o400->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788580420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10820.609703] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10824.113858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10830.815962] LustreError: 340514:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002c2a23d5 x1875462127196032/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10831.177972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10831.247249] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10833.486877] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10836.469499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10836.495445] Lustre: Skipped 2 previous similar messages [10836.522217] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10836.589536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10836.976980] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10848.697441] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10850.584682] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10856.932840] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10856.936824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10856.940727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10856.957967] Lustre: Skipped 1 previous similar message [10856.985598] Lustre: Skipped 4 previous similar messages [10862.605953] Lustre: server umount lustre-MDT0000 complete [10867.777657] LustreError: 347003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580467 with bad export cookie 18337161109035392012 [10867.792137] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10867.793328] LustreError: 347003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10868.292205] Lustre: server umount lustre-MDT0001 complete [10884.098993] Lustre: server umount lustre-OST0000 complete [10902.112746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10902.609145] LustreError: 353388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10902.647086] LustreError: 353388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [10902.715331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10908.003585] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10918.294623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10924.305485] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10933.329920] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10940.260250] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10947.556882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10955.242331] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10964.401401] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10972.466464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10982.882584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10982.912839] Lustre: Skipped 2 previous similar messages [10982.931636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10982.941081] Lustre: Skipped 4 previous similar messages [10987.079962] Lustre: server umount lustre-OST0000 complete [10997.704188] Lustre: server umount lustre-MDT0000 complete [11001.372815] LustreError: 353369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580601 with bad export cookie 18337161109035392761 [11001.382901] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11001.385188] LustreError: 353369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11001.716446] Lustre: server umount lustre-MDT0001 complete [11013.505377] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [11016.651866] Key type lgssc unregistered [11017.016725] LNet: 356947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11017.031422] LNetError: 356947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11017.066666] LNet: Removed LNI 192.168.204.154@tcp [11018.135920] Key type .llcrypt unregistered [11018.138918] Key type ._llcrypt unregistered [11047.598364] Key type ._llcrypt registered [11047.603984] Key type .llcrypt registered [11047.776849] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11049.382560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11049.432333] alg: No test for adler32 (adler32-zlib) [11050.578246] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [11050.889448] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [11052.641539] Key type lgssc registered [11054.091428] Lustre: Echo OBD driver; http://www.lustre.org/ [11062.818849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11062.849130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11064.278283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11069.594791] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11074.547748] LustreError: 358466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11074.563726] LustreError: 358466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11079.650337] LustreError: 358467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11079.749611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11080.196749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11084.929056] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11092.824767] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11098.501887] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11106.623610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11107.153307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11109.251413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11114.678430] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11122.426874] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11138.468294] Lustre: server umount lustre-MDT0000 complete [11141.095331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11141.107872] LustreError: 360876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11141.111021] Lustre: Skipped 1 previous similar message [11141.154845] LustreError: 360876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11142.741643] LustreError: 358448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580742 with bad export cookie 16679632523709692350 [11142.751178] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11142.762109] LustreError: 358448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11143.170846] Lustre: server umount lustre-MDT0001 complete [11158.168591] Lustre: server umount lustre-OST0000 complete [11170.922832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11171.375267] LustreError: 362201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11171.433737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11175.679922] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11176.934718] LustreError: 362202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11184.100680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11184.508670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11189.121598] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11197.759293] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11204.499413] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11211.753470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11212.036528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11216.117897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11219.134173] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11228.396657] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11240.929293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11240.943842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11240.967024] Lustre: Skipped 1 previous similar message [11240.981401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11245.029446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11245.035090] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11246.727169] Lustre: server umount lustre-MDT0000 complete [11250.147110] LustreError: 362196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11250.184148] LustreError: 362196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11251.444020] LustreError: 362183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580851 with bad export cookie 16679632523709693463 [11251.450791] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11251.465797] LustreError: 362183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11251.825867] Lustre: server umount lustre-MDT0001 complete [11267.190299] Lustre: server umount lustre-OST0000 complete [11284.205112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11284.784174] LustreError: 366129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11284.879641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11289.432618] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11299.071024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11299.296953] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11299.313477] Lustre: Skipped 2 previous similar messages [11303.616116] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11313.209685] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11320.669862] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11330.738113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11331.181963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11331.191417] Lustre: Skipped 1 previous similar message [11338.616636] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11347.027407] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11354.759397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11365.348826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11365.362116] Lustre: Skipped 2 previous similar messages [11365.362878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11365.376584] Lustre: Skipped 1 previous similar message [11370.466455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11370.476532] Lustre: Skipped 1 previous similar message [11377.119214] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11377.252604] Lustre: server umount lustre-OST0000 complete [11381.732622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11381.740640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11381.747974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11381.757177] Lustre: Skipped 2 previous similar messages [11395.039937] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11395.410413] Lustre: server umount lustre-MDT0000 complete [11396.071276] LustreError: 368622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11396.109184] LustreError: 368622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11400.599426] LustreError: 366111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581000 with bad export cookie 16679632523709694485 [11400.621059] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11401.326337] Lustre: server umount lustre-MDT0001 complete [11413.275652] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [11416.061819] Key type lgssc unregistered [11416.432046] LNet: 369689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11416.443870] LNetError: 369689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11416.464101] LNet: Removed LNI 192.168.204.154@tcp [11417.383268] Key type .llcrypt unregistered [11417.384944] Key type ._llcrypt unregistered [11450.256393] Key type ._llcrypt registered [11450.260993] Key type .llcrypt registered [11450.432961] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11452.277811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11452.353385] alg: No test for adler32 (adler32-zlib) [11453.505802] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [11453.765526] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [11455.423193] Key type lgssc registered [11457.091629] Lustre: Echo OBD driver; http://www.lustre.org/ [11466.857360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11466.876465] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11466.885683] LustreError: 371133:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11466.911603] LustreError: 371133:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11466.923181] LustreError: 371133:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11466.938686] LustreError: 371133:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11466.951590] LustreError: 371133:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11475.198183] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11475.207754] LustreError: 371532:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11475.226353] LustreError: 371532:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11475.236734] LustreError: 371532:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11475.242709] LustreError: 371532:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11475.250730] LustreError: 371532:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11495.312661] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [11499.143573] Key type lgssc unregistered [11499.659718] LNet: 372538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11499.666688] LNetError: 372538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11499.695633] LNet: Removed LNI 192.168.204.154@tcp [11501.089541] Key type .llcrypt unregistered [11501.094504] Key type ._llcrypt unregistered [11524.005329] Key type ._llcrypt registered [11524.010860] Key type .llcrypt registered [11524.147705] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [11540.172249] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11541.331408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11541.405024] alg: No test for adler32 (adler32-zlib) [11542.486833] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [11542.802984] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [11544.511365] Key type lgssc registered [11545.445678] Lustre: Echo OBD driver; http://www.lustre.org/ [11588.046212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11588.110213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11589.531473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11589.581821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11589.678768] Lustre: lustre-MDT0000: new disk, initializing [11589.787649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11589.857420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11595.158454] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11605.706852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11605.819485] Lustre: 375696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11605.882553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11605.887392] Lustre: Skipped 1 previous similar message [11605.970504] Lustre: lustre-MDT0001: new disk, initializing [11606.049253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11606.074862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11606.094125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11612.098788] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11625.578850] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11632.824533] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11641.246458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11641.564904] Lustre: lustre-OST0000: new disk, initializing [11641.568948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11641.578509] Lustre: 377667:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11641.678697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11649.110035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11649.124764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11649.262412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11649.578389] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11661.125359] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11666.788850] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11667.020415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11671.641099] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11671.914125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11675.106280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11675.107651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11675.128509] LustreError: Skipped 1 previous similar message [11675.133348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11675.147279] Lustre: Skipped 1 previous similar message [11675.169290] Lustre: Skipped 1 previous similar message [11679.999681] Lustre: server umount lustre-OST0000 complete [11685.351956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11685.356959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11685.364315] Lustre: Skipped 1 previous similar message [11685.383459] Lustre: Skipped 1 previous similar message [11690.468829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11690.486353] Lustre: Skipped 1 previous similar message [11695.585805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11698.145067] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11698.321628] Lustre: server umount lustre-MDT0000 complete [11702.707291] LustreError: 375687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581302 with bad export cookie 9667022574198158257 [11702.722961] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11703.084141] Lustre: server umount lustre-MDT0001 complete [11715.341365] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11718.358449] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:08:35 (1788581315) [11726.216647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11726.777847] LustreError: 380131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11726.799090] LustreError: 380131:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11726.849572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11731.525676] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11731.943640] LustreError: 380132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11737.058413] LustreError: 380131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11740.328086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11740.805192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11746.106291] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11754.807258] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11761.673609] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11770.509085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11770.932471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11779.406911] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11789.488825] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11798.411515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11798.627793] Lustre: lustre-OST0001: new disk, initializing [11798.632767] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11798.638783] Lustre: 382971:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11798.741208] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11804.302152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11804.326525] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11804.407099] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11806.980797] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11817.932140] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11825.121910] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11825.128173] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11825.135471] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11825.135483] Lustre: Skipped 1 previous similar message [11825.154164] Lustre: Skipped 1 previous similar message [11828.628079] Lustre: server umount lustre-OST0001 complete [11830.253215] LustreError: 381953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11830.272832] LustreError: 381953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11835.367696] LustreError: 381954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11835.384987] LustreError: 381954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11845.625820] LustreError: 381954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11845.640850] LustreError: 381954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11848.679550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11848.695422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11848.729241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11848.737935] Lustre: Skipped 2 previous similar messages [11853.166458] Lustre: server umount lustre-OST0000 complete [11858.402531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11858.410831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11858.423366] Lustre: Skipped 1 previous similar message [11866.093914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11866.104593] Lustre: Skipped 6 previous similar messages [11872.223285] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11872.445815] Lustre: server umount lustre-MDT0000 complete [11875.301467] LustreError: 382013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11875.317932] LustreError: 382013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11876.805211] LustreError: 382970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581476 with bad export cookie 9667022574198159139 [11876.808225] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11876.818554] LustreError: 382970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11891.168828] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11891.665765] Lustre: server umount lustre-MDT0001 complete [11904.485083] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [11908.151472] Key type lgssc unregistered [11908.642686] LNet: 384971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11908.651627] LNetError: 384971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11908.695637] LNet: Removed LNI 192.168.204.154@tcp [11909.625716] Key type .llcrypt unregistered [11909.628269] Key type ._llcrypt unregistered [11970.144858] Key type ._llcrypt registered [11970.148387] Key type .llcrypt registered [11970.273856] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11972.111749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11972.176766] alg: No test for adler32 (adler32-zlib) [11973.210092] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [11973.468513] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [11975.311961] Key type lgssc registered [11976.592507] Lustre: Echo OBD driver; http://www.lustre.org/ [11986.826501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11986.854093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11988.267450] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11988.304110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11988.601309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11993.492740] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12004.655235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12004.815977] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12004.848171] Lustre: 387769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12005.349098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12011.136421] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12021.596375] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12029.161863] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12033.510246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12033.518265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12033.523969] Lustre: Skipped 1 previous similar message [12037.857777] Lustre: server umount lustre-MDT0000 complete [12042.257527] LustreError: 388514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581642 with bad export cookie 7820022256125106576 [12042.264626] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12042.272858] LustreError: 388514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12042.645245] Lustre: server umount lustre-MDT0001 complete [12053.890697] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:14:11 (1788581651) [12061.881129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12062.413187] LustreError: 390287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12062.437391] LustreError: 390287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12062.482885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12067.564939] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12067.815206] LustreError: 390288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12072.929934] LustreError: 390287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12077.039424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12077.564276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12085.927164] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12098.192359] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12106.137859] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12114.319802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12114.548079] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12114.813121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12122.356240] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12131.147685] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12139.300541] Lustre: Permanently deactivating lustre-OST0000 [12139.321264] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12145.212370] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12148.059924] LustreError: 393227:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12148.069510] LustreError: 393227:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12148.075895] LustreError: 393227:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12151.658498] LustreError: 393276:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12151.665524] LustreError: 393276:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12159.670250] Lustre: server umount lustre-OST0000 complete [12162.449373] LustreError: 393526:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12162.455242] LustreError: 393526:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12169.696474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12169.699837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12169.705586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12169.730175] Lustre: Skipped 2 previous similar messages [12169.745151] Lustre: Skipped 1 previous similar message [12174.818028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12174.824132] Lustre: Skipped 1 previous similar message [12175.957572] Lustre: server umount lustre-MDT0000 complete [12179.937362] LustreError: 391013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12180.901639] LustreError: 390267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581780 with bad export cookie 7820022256125107157 [12180.902206] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12180.915613] LustreError: 390267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12190.432457] Lustre: server umount lustre-MDT0001 complete [12201.142905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12207.069431] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12210.711727] LustreError: 394690:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12218.754160] Lustre: 395029:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.154@tcp [12221.409613] Lustre: server umount MGS complete [12235.071519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12235.699572] LustreError: 395754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12235.719356] LustreError: 395754:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12235.722379] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12235.732382] Lustre: Skipped 1 previous similar message [12235.781431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12241.194898] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12245.984377] LustreError: 395755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12246.010063] LustreError: 395755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12251.388962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12251.710072] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12251.792908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12256.989296] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12266.174471] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12273.295261] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12275.637972] Lustre: Permanently reactivating lustre-OST0000 [12275.650250] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12275.655541] Lustre: Skipped 2 previous similar messages [12304.352116] LustreError: 397536:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12311.659323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12312.020359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12318.571092] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12326.734078] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12333.537050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12333.552465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12333.566430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12334.573829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12339.685125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12339.693128] Lustre: Skipped 3 previous similar messages [12347.360069] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12347.679979] Lustre: server umount lustre-OST0000 complete [12354.016765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12354.029392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12354.044991] Lustre: Skipped 1 previous similar message [12354.051648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12354.059709] Lustre: Skipped 2 previous similar messages [12366.303493] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12366.537452] Lustre: server umount lustre-MDT0000 complete [12370.400799] LustreError: 397196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12370.424205] LustreError: 397196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12370.451035] LustreError: 395734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581970 with bad export cookie 7820022256125108200 [12370.453093] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12370.460428] LustreError: 395734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12370.960419] Lustre: server umount lustre-MDT0001 complete [12382.513354] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12385.374477] Key type lgssc unregistered [12385.754951] LNet: 400126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12385.764040] LNetError: 400126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12385.787357] LNet: Removed LNI 192.168.204.154@tcp [12386.704648] Key type .llcrypt unregistered [12386.708536] Key type ._llcrypt unregistered [12409.221332] Key type ._llcrypt registered [12409.223773] Key type .llcrypt registered [12409.360400] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [12427.504547] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [12428.759278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12428.929735] alg: No test for adler32 (adler32-zlib) [12430.211922] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [12430.525757] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [12432.327236] Key type lgssc registered [12433.839725] Lustre: Echo OBD driver; http://www.lustre.org/ [12480.101248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12480.142141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12481.546640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12481.642152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12481.778965] Lustre: lustre-MDT0000: new disk, initializing [12481.956856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12482.000189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12487.185507] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12500.597537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12500.752557] Lustre: 403289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12500.841047] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12500.847474] Lustre: Skipped 1 previous similar message [12500.943796] Lustre: lustre-MDT0001: new disk, initializing [12501.087965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12501.121665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12501.141884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12505.807320] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12516.950458] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12524.198516] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12531.446129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12531.772785] Lustre: lustre-OST0000: new disk, initializing [12531.776361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12531.782446] Lustre: 405258:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12531.864844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12538.519380] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12541.544112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12541.565020] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12541.684048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12549.747520] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12556.654708] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12557.030733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12562.647737] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12562.821348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12567.523468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12567.528450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12567.533356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12567.537149] Lustre: Skipped 1 previous similar message [12572.641224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12572.665409] Lustre: Skipped 2 previous similar messages [12577.779446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12577.804721] Lustre: Skipped 1 previous similar message [12579.296183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12579.720654] Lustre: server umount lustre-OST0000 complete [12588.002125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12588.004634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12588.009550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12588.017647] LustreError: Skipped 1 previous similar message [12590.535856] Lustre: server umount lustre-MDT0000 complete [12594.405079] LustreError: 403282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582194 with bad export cookie 14568012506873159434 [12594.418108] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12594.418354] LustreError: 403282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12594.751183] Lustre: server umount lustre-MDT0001 complete [12605.715791] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12607.517453] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:23:25 (1788582205) [12614.645300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12615.321540] LustreError: 407723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12615.335708] LustreError: 407723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12615.398954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12620.441365] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12620.770866] LustreError: 407724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12625.892536] LustreError: 407723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12629.759800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12630.094226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12635.055488] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12642.993714] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12649.607251] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12658.130453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12658.651912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12666.685879] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12674.894635] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12677.074730] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12679.729442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12686.319425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12686.334878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12686.352710] Lustre: Skipped 1 previous similar message [12686.364614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12686.385061] Lustre: Skipped 1 previous similar message [12689.387297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12689.397276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12691.505265] Lustre: server umount lustre-OST0000 complete [12696.544376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12696.551282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12696.567778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12696.575220] Lustre: Skipped 2 previous similar messages [12701.939280] Lustre: server umount lustre-MDT0000 complete [12704.737578] LustreError: 408451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12704.764274] LustreError: 408451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12705.788033] LustreError: 408459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582305 with bad export cookie 14568012506873160316 [12705.803289] LustreError: 408459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12705.804833] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12706.218883] Lustre: server umount lustre-MDT0001 complete [12718.282572] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12721.328831] Key type lgssc unregistered [12721.617296] LNet: 411452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12721.627960] LNetError: 411452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12721.651408] LNet: Removed LNI 192.168.204.154@tcp [12722.615371] Key type .llcrypt unregistered [12722.618482] Key type ._llcrypt unregistered [12749.221072] Key type ._llcrypt registered [12749.224151] Key type .llcrypt registered [12749.405547] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12749.869375] Key type .llcrypt unregistered [12749.880599] Key type ._llcrypt unregistered [12770.233335] Key type ._llcrypt registered [12770.236562] Key type .llcrypt registered [12770.377378] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [12771.900703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12771.953078] alg: No test for adler32 (adler32-zlib) [12773.063398] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [12773.329522] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [12775.016506] Key type lgssc registered [12776.563426] Lustre: Echo OBD driver; http://www.lustre.org/ [12785.525619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12785.563232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12787.181438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12792.034817] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12797.413573] LustreError: 413673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12797.437899] LustreError: 413673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12800.628886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12800.971174] LustreError: 413674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12800.993320] LustreError: 413674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12801.002508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12806.118427] LustreError: 413673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12806.155571] LustreError: 413673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12807.933849] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12811.244320] LustreError: 413669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12811.296861] LustreError: 413669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12812.279991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12817.099836] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12817.382309] LustreError: 413669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12817.402859] LustreError: 413669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12826.028060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12826.588909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12832.620427] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12842.979264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12842.989128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12843.007284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12847.076070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12847.077600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12847.085853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12852.197678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12852.217326] Lustre: Skipped 1 previous similar message [12856.287112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12856.584640] Lustre: server umount lustre-OST0000 complete [12862.432419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12862.434299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12862.480724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12862.494461] Lustre: Skipped 1 previous similar message [12862.502217] Lustre: Skipped 1 previous similar message [12867.040657] Lustre: server umount lustre-MDT0000 complete [12867.555953] LustreError: 413669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12867.576045] LustreError: 413669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12871.013340] LustreError: 413653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582470 with bad export cookie 7453019669947269179 [12871.028740] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12871.348663] Lustre: server umount lustre-MDT0001 complete [12883.344385] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12887.093465] Key type lgssc unregistered [12887.558326] LNet: 416826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12887.568400] LNetError: 416826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12887.587780] LNet: Removed LNI 192.168.204.154@tcp [12888.662640] Key type .llcrypt unregistered [12888.666905] Key type ._llcrypt unregistered [12914.688418] Key type ._llcrypt registered [12914.689961] Key type .llcrypt registered [12914.782164] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [12915.609654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12915.780714] alg: No test for adler32 (adler32-zlib) [12916.871773] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [12917.137085] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [12918.816599] Key type lgssc registered [12919.963522] Lustre: Echo OBD driver; http://www.lustre.org/ [12927.786486] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12927.807940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12943.391455] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12948.511329] LustreError: 418216:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [12975.199332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12983.195446] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12992.639480] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12999.173846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12999.706218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12999.789943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13003.425642] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13004.778158] LustreError: 418241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13004.803491] LustreError: 418241:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13009.892147] LustreError: 418242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13009.930908] LustreError: 418242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13013.378619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13013.844804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13018.617231] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13027.054730] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13033.572272] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13044.704944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13044.738202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13047.992091] Lustre: server umount lustre-OST0000 complete [13054.945648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13054.951862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13054.956191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13054.956199] Lustre: Skipped 1 previous similar message [13054.979333] Lustre: Skipped 2 previous similar messages [13058.717316] Lustre: server umount lustre-MDT0000 complete [13060.066113] LustreError: 419254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13060.097125] LustreError: 419254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13063.595020] LustreError: 418224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582663 with bad export cookie 16248496156032523349 [13063.602814] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13063.617436] LustreError: 418224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13063.931753] Lustre: server umount lustre-MDT0001 complete [13077.266895] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13080.651573] Key type lgssc unregistered [13081.093775] LNet: 421788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13081.104678] LNetError: 421788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13081.140974] LNet: Removed LNI 192.168.204.154@tcp [13082.205955] Key type .llcrypt unregistered [13082.208163] Key type ._llcrypt unregistered [13111.884353] Key type ._llcrypt registered [13111.888158] Key type .llcrypt registered [13111.985121] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [13113.143723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13113.228145] alg: No test for adler32 (adler32-zlib) [13114.611767] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [13114.934121] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [13116.688345] Key type lgssc registered [13117.892186] Lustre: Echo OBD driver; http://www.lustre.org/ [13125.839845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13125.866807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13127.370985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13132.388709] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13137.888627] LustreError: 423207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13137.903120] LustreError: 423207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13141.324669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13142.087044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13148.788317] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13157.944910] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13164.185266] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13172.945216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13173.391758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13181.103657] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13190.568660] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13198.005393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13207.651327] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [13208.550344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13208.556376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13208.559892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13208.570478] Lustre: Skipped 2 previous similar messages [13208.577150] Lustre: Skipped 1 previous similar message [13212.259474] Lustre: server umount lustre-MDT0000 complete [13212.750216] LustreError: 423204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13215.456884] LustreError: 423232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.54@tcp arrived at 1788582815 with bad export cookie 14960542103495000742 [13215.478640] LustreError: 423232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13218.159612] LustreError: 423188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582818 with bad export cookie 14960542103495000077 [13218.176094] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13218.786217] LustreError: 423943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13218.790768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13218.823824] LustreError: 423943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13218.874291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13218.876712] Lustre: Skipped 1 previous similar message [13223.905602] LustreError: 423936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13223.907569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13223.933765] LustreError: 423936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13224.588989] Lustre: server umount lustre-OST0000 complete [13229.024409] LustreError: 423208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13229.050775] LustreError: 423208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13233.415316] Lustre: server umount lustre-MDT0001 complete [13246.278343] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13249.949829] Key type lgssc unregistered [13250.347853] LNet: 426918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13250.358909] LNetError: 426918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13250.398771] LNet: Removed LNI 192.168.204.154@tcp [13251.600420] Key type .llcrypt unregistered [13251.606122] Key type ._llcrypt unregistered [13280.427428] Key type ._llcrypt registered [13280.429972] Key type .llcrypt registered [13280.529844] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [13281.649661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13281.668803] alg: No test for adler32 (adler32-zlib) [13282.769810] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [13283.050765] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [13284.719174] Key type lgssc registered [13286.429973] Lustre: Echo OBD driver; http://www.lustre.org/ [13295.128483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13295.163509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13296.778069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13302.610702] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13306.340715] LustreError: 428336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13306.389710] LustreError: 428336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13311.457526] LustreError: 428335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13312.715606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13313.397550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13318.255526] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13326.658216] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13333.434787] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13339.875526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13340.217767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13341.239869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13347.520125] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13355.775649] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13363.682441] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13363.705163] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13363.714542] Lustre: Skipped 1 previous similar message [13366.914060] Lustre: server umount lustre-MDT0001 complete [13368.804874] LustreError: 428336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13368.819920] LustreError: 428336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13373.922729] LustreError: 429339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13373.924636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13373.941296] LustreError: 429339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13373.977193] Lustre: Skipped 1 previous similar message [13373.985654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13374.001303] Lustre: Skipped 1 previous similar message [13379.041713] LustreError: 428335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13379.042166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13379.971960] Lustre: server umount lustre-OST0000 complete [13385.274887] Lustre: server umount lustre-MDT0000 complete [13399.638448] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13402.689379] Key type lgssc unregistered [13403.035316] LNet: 432023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13403.052977] LNetError: 432023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13403.073379] LNet: Removed LNI 192.168.204.154@tcp [13404.127291] Key type .llcrypt unregistered [13404.128898] Key type ._llcrypt unregistered [13432.622243] Key type ._llcrypt registered [13432.625261] Key type .llcrypt registered [13432.742549] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [13449.231055] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [13450.930285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13450.965830] alg: No test for adler32 (adler32-zlib) [13452.005560] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [13452.249211] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [13453.919248] Key type lgssc registered [13455.439162] Lustre: Echo OBD driver; http://www.lustre.org/ [13498.536388] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [13518.281222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13518.327038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13519.748257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13519.778240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13519.836290] Lustre: lustre-MDT0000: new disk, initializing [13519.952084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13519.976428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13524.629551] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13542.064287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13542.317319] Lustre: lustre-OST0000: new disk, initializing [13542.323517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13542.328651] Lustre: Skipped 1 previous similar message [13542.340668] Lustre: 437064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13542.472582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13549.486578] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13551.156479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13551.172631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13551.223552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13559.456796] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13576.491116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13576.670940] Lustre: 436046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13576.760232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13576.862289] Lustre: lustre-MDT0001: new disk, initializing [13577.009544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13577.067349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13577.080080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13582.684694] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13593.900980] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13594.120464] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13599.629137] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13599.894759] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13606.371560] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13606.383796] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13606.404724] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13606.412646] Lustre: Skipped 1 previous similar message [13606.430873] Lustre: Skipped 1 previous similar message [13607.004523] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [13611.325228] Lustre: server umount lustre-MDT0001 complete [13616.617103] LustreError: 436055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13616.640052] LustreError: 436055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13621.734288] LustreError: 436052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13621.749981] LustreError: 436052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13626.859098] LustreError: 436056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13626.880638] LustreError: 436056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13631.970038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13631.972047] LustreError: 436055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13631.998033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13635.538660] Lustre: server umount lustre-OST0000 complete [13637.107170] LustreError: 436056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13646.086727] Lustre: server umount lustre-MDT0000 complete [13660.462944] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13663.745645] Key type lgssc unregistered [13664.177729] LNet: 440811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13664.190766] LNetError: 440811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13665.254894] LNet: Removed LNI 192.168.204.154@tcp [13666.116232] Key type .llcrypt unregistered [13666.120093] Key type ._llcrypt unregistered [13710.287443] Key type ._llcrypt registered [13710.290875] Key type .llcrypt registered [13710.448587] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13710.886375] Key type .llcrypt unregistered [13710.894457] Key type ._llcrypt unregistered [13726.436281] Key type ._llcrypt registered [13726.439700] Key type .llcrypt registered [13726.582565] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [13728.077095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13728.175846] alg: No test for adler32 (adler32-zlib) [13729.288615] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [13729.569172] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [13731.255459] Key type lgssc registered [13732.871544] Lustre: Echo OBD driver; http://www.lustre.org/ [13763.407271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13763.430357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13764.834498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13764.860708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13764.971903] Lustre: lustre-MDT0000: new disk, initializing [13765.058675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13765.071873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13770.849485] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13782.323547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13782.490354] Lustre: 444194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13782.536581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13782.539612] Lustre: Skipped 1 previous similar message [13782.595589] Lustre: lustre-MDT0001: new disk, initializing [13782.716913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13782.768970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13782.776591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13787.864951] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13799.298985] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13806.320533] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13814.597291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13814.983952] Lustre: lustre-OST0000: new disk, initializing [13814.987973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13814.995918] Lustre: 446165:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13815.126314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13821.531437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13821.545435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13821.671837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13822.334782] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13833.370841] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13842.404046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13842.409526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13842.421480] Lustre: Skipped 1 previous similar message [13842.437746] Lustre: Skipped 2 previous similar messages [13846.727160] Lustre: server umount lustre-MDT0000 complete [13851.005617] LustreError: 444186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583450 with bad export cookie 8064320316294558579 [13851.007146] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13851.023712] LustreError: 444186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13851.365672] Lustre: server umount lustre-MDT0001 complete [13866.163244] Lustre: server umount lustre-OST0000 complete [13876.916172] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:44:35 (1788583475) [13884.985819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13886.263507] LustreError: 448194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13886.293546] LustreError: 448194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13886.458257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13891.552912] LustreError: 448195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13893.283530] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13896.672863] LustreError: 448194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13901.796894] LustreError: 448195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13904.260550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13904.780074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13910.418753] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13920.306840] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13926.838663] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13947.311618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13947.558937] Lustre: Found index 0 for lustre-OST0000, updating log [13947.806542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13954.614373] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13958.060372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13963.070058] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13968.359623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13968.368353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13968.371526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13968.392433] Lustre: Skipped 2 previous similar messages [13969.457825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13969.466625] Lustre: Skipped 2 previous similar messages [13974.093061] Lustre: server umount lustre-OST0000 complete [13990.880701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13990.905547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13995.746372] Lustre: server umount lustre-MDT0000 complete [13996.002880] LustreError: 449881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13996.018794] LustreError: 449881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13999.549017] LustreError: 448175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583599 with bad export cookie 8064320316294559965 [13999.553513] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13999.560722] LustreError: 448175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13999.992244] Lustre: server umount lustre-MDT0001 complete [14005.087441] LNet: 1 local NIs in recovery (showing 1): 192.168.204.154@tcp [14008.789386] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:46:46 (1788583606) [14626.681272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14627.131052] LustreError: 454604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14627.155123] LustreError: 454604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14627.228105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14631.053252] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [14632.420961] LustreError: 454605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14637.540668] LustreError: 454604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14639.292545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14639.716841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14644.277407] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [14651.815894] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14657.306509] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14677.415218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14677.674580] Lustre: Found index 0 for lustre-OST0000, updating log [14678.053784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14683.999307] LNet: 1 local NIs in recovery (showing 1): 192.168.204.154@tcp [14684.373038] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [14691.697742] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14693.847569] Lustre: server umount lustre-OST0000 complete [14697.392669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14697.402433] Lustre: Skipped 1 previous similar message [14697.403952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14697.409889] Lustre: Skipped 1 previous similar message [14698.465616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14698.485822] Lustre: Skipped 2 previous similar messages [14699.491711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14699.496648] Lustre: Skipped 1 previous similar message [14701.540849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14701.546655] Lustre: Skipped 3 previous similar messages [14703.719799] Lustre: server umount lustre-MDT0000 complete [14704.615278] LustreError: 454629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14704.630677] LustreError: 454629:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14707.474654] LustreError: 454585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584307 with bad export cookie 8064320316294561057 [14707.480617] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14707.508917] LustreError: 454585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14707.862259] Lustre: server umount lustre-MDT0001 complete [14737.788667] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [14746.301163] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [14747.491437] LNet: 1 local NIs in recovery (showing 1): 192.168.204.154@tcp [14777.292701] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 00:59:35 (1788584375) [14796.421958] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [14799.321554] Key type lgssc unregistered [14799.709660] LNet: 461211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14799.719624] LNetError: 461211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14890.875382] LNet: Removed LNI 192.168.204.154@tcp [14891.672908] Key type .llcrypt unregistered [14891.683054] Key type ._llcrypt unregistered [14907.019301] Key type ._llcrypt registered [14907.024949] Key type .llcrypt registered [14907.148558] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [14907.954293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14908.003360] alg: No test for adler32 (adler32-zlib) [14909.071220] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [14909.316291] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [14911.055138] Key type lgssc registered [14912.400282] Lustre: Echo OBD driver; http://www.lustre.org/ [14916.863811] LNet: Added LNI 192.168.204.154@tcp100 [8/256/0/180] [14917.662468] LNet: Added LNI 192.168.204.154@tcp101 [8/256/0/180] [14918.415487] LNet: Added LNI 192.168.204.154@tcp102 [8/256/0/180] [14919.150102] LNet: Added LNI 192.168.204.154@tcp103 [8/256/0/180] [14919.940964] LNet: Added LNI 192.168.204.154@tcp104 [8/256/0/180] [14920.739435] LNet: Added LNI 192.168.204.154@tcp105 [8/256/0/180] [14921.505217] LNet: Added LNI 192.168.204.154@tcp106 [8/256/0/180] [14922.348779] LNet: Added LNI 192.168.204.154@tcp107 [8/256/0/180] [14923.118443] LNet: Added LNI 192.168.204.154@tcp108 [8/256/0/180] [14923.997036] LNet: Added LNI 192.168.204.154@tcp109 [8/256/0/180] [14924.751073] LNet: Added LNI 192.168.204.154@tcp110 [8/256/0/180] [14925.634778] LNet: Added LNI 192.168.204.154@tcp111 [8/256/0/180] [14926.478743] LNet: Added LNI 192.168.204.154@tcp112 [8/256/0/180] [14927.356438] LNet: Added LNI 192.168.204.154@tcp113 [8/256/0/180] [14928.103968] LNet: Added LNI 192.168.204.154@tcp114 [8/256/0/180] [14928.876302] LNet: Added LNI 192.168.204.154@tcp115 [8/256/0/180] [14929.835311] LNet: Added LNI 192.168.204.154@tcp116 [8/256/0/180] [14930.729456] LNet: Added LNI 192.168.204.154@tcp117 [8/256/0/180] [14931.532894] LNet: Added LNI 192.168.204.154@tcp118 [8/256/0/180] [14932.354466] LNet: Added LNI 192.168.204.154@tcp119 [8/256/0/180] [14933.136986] LNet: Added LNI 192.168.204.154@tcp120 [8/256/0/180] [14933.966156] LNet: Added LNI 192.168.204.154@tcp121 [8/256/0/180] [14934.867945] LNet: Added LNI 192.168.204.154@tcp122 [8/256/0/180] [14935.621983] LNet: Added LNI 192.168.204.154@tcp123 [8/256/0/180] [14936.339695] LNet: Added LNI 192.168.204.154@tcp124 [8/256/0/180] [14937.198545] LNet: Added LNI 192.168.204.154@tcp125 [8/256/0/180] [14937.788185] LNet: Added LNI 192.168.204.154@tcp126 [8/256/0/180] [14938.843296] LNet: Added LNI 192.168.204.154@tcp127 [8/256/0/180] [14939.689121] LNet: Added LNI 192.168.204.154@tcp128 [8/256/0/180] [14940.442903] LNet: Added LNI 192.168.204.154@tcp129 [8/256/0/180] [14941.028780] LNet: Added LNI 192.168.204.154@tcp130 [8/256/0/180] [14941.945889] LNet: Added LNI 192.168.204.154@tcp131 [8/256/0/180] [14942.651574] LNet: Added LNI 192.168.204.154@tcp132 [8/256/0/180] [14943.264355] LNet: Added LNI 192.168.204.154@tcp133 [8/256/0/180] [14944.124418] LNet: Added LNI 192.168.204.154@tcp134 [8/256/0/180] [14945.108713] LNet: Added LNI 192.168.204.154@tcp135 [8/256/0/180] [14951.931534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14951.990746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14953.097691] Lustre: 464254:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [14953.206291] LNetError: 461713:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.154 [14953.225839] LNetError: Protocol error connecting to 192.168.204.154@tcp114 at host 192.168.204.154:988: is it running a compatible version of Lustre? [14953.236075] LNetError: Skipped 5 previous similar messages [14953.313133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14953.344322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14953.375190] LNet: 5 local NIs in recovery (showing 5): 192.168.204.154@tcp123, 192.168.204.154@tcp124, 192.168.204.154@tcp125, 192.168.204.154@tcp121, 192.168.204.154@tcp122 [14953.450201] Lustre: lustre-MDT0000: new disk, initializing [14953.552647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14953.589773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14954.273957] LNetError: 461713:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.154 [14954.289215] LNetError: 461713:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [14954.303419] LNetError: Protocol error connecting to 192.168.204.154@tcp103 at host 192.168.204.154:988: is it running a compatible version of Lustre? [14954.319437] LNetError: Skipped 1 previous similar message [14956.450197] LNetError: 461714:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.154 [14956.478956] LNetError: 461714:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [14956.494489] LNetError: Protocol error connecting to 192.168.204.154@tcp123 at host 192.168.204.154:988: is it running a compatible version of Lustre? [14956.509607] LNetError: Skipped 5 previous similar messages [14957.955761] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [14960.549055] LNetError: 464276:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.154 [14960.572559] LNetError: Protocol error connecting to 192.168.204.154@tcp123 at host 192.168.204.154:988: is it running a compatible version of Lustre? [14967.999540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14968.127204] Lustre: 464269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14968.183290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14968.186543] Lustre: Skipped 1 previous similar message [14968.264876] Lustre: lustre-MDT0001: new disk, initializing [14968.307557] Lustre: 465075:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [14968.315254] Lustre: 465075:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14968.401488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14968.433894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14968.441138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14968.743200] LNetError: 464282:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.154 [14968.759078] LNetError: Protocol error connecting to 192.168.204.154@tcp123 at host 192.168.204.154:988: is it running a compatible version of Lustre? [14969.482468] Lustre: 465153:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [14969.491812] Lustre: 465153:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14972.785240] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [14981.270103] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14983.073299] LNetError: 461715:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.154 [14983.090517] LNetError: Protocol error connecting to 192.168.204.154@tcp103 at host 192.168.204.154:988: is it running a compatible version of Lustre? [14986.461700] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14992.517731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14992.850872] Lustre: lustre-OST0000: new disk, initializing [14992.855171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14992.862778] Lustre: 466246:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14992.963775] Lustre: 466251:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [14992.968882] Lustre: 466251:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14993.007251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14998.061125] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [14999.073315] Lustre: 466656:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [14999.079149] Lustre: 466656:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14999.172362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14999.188898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14999.345191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14999.459484] LNetError: 464276:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.154 [14999.466826] LNetError: 464276:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [14999.472436] LNetError: Protocol error connecting to 192.168.204.154@tcp114 at host 192.168.204.154:988: is it running a compatible version of Lustre? [14999.489874] LNetError: Skipped 8 previous similar messages [15006.953702] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15013.796365] LNet: 1 local NIs in recovery (showing 1): 192.168.204.154@tcp123 [15014.880877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15014.884276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15014.895105] LustreError: Skipped 1 previous similar message [15014.900859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15014.914137] Lustre: Skipped 1 previous similar message [15014.929056] Lustre: Skipped 1 previous similar message [15019.990374] Lustre: server umount lustre-OST0000 complete [15024.608492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15024.615776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15024.629718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15030.243220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15030.255302] Lustre: Skipped 3 previous similar messages [15030.413994] Lustre: server umount lustre-MDT0000 complete [15034.531993] LustreError: 464262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584634 with bad export cookie 3438315987444000892 [15034.538396] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15034.541549] LustreError: 464262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15034.856966] Lustre: server umount lustre-MDT0001 complete [15035.296679] LNetError: 464274:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.154 [15035.304548] LNetError: 464274:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 15 previous similar messages [15035.322146] LNetError: Protocol error connecting to 192.168.204.154@tcp103 at host 192.168.204.154:988: is it running a compatible version of Lustre? [15035.342421] LNetError: Skipped 15 previous similar messages [15047.903922] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [15051.096186] Key type lgssc unregistered [15051.541034] LNet: 468165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15051.547892] LNetError: 468165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15051.554202] LNet: Removed LNI 192.168.204.154@tcp [15052.576564] LNet: Removed LNI 192.168.204.154@tcp100 [15052.601933] LNetError: 468165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15052.613733] LNetError: 468165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15052.624284] LNet: Removed LNI 192.168.204.154@tcp101 [15053.663327] LNet: Removed LNI 192.168.204.154@tcp102 [15053.670363] LNetError: 468165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15053.688382] LNetError: 468165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15053.705969] LNet: Removed LNI 192.168.204.154@tcp103 [15054.751397] LNet: Removed LNI 192.168.204.154@tcp104 [15055.775208] LNet: Removed LNI 192.168.204.154@tcp105 [15055.779936] LNetError: 468165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15055.793109] LNetError: 468165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15055.803568] LNet: Removed LNI 192.168.204.154@tcp106 [15056.863209] LNet: Removed LNI 192.168.204.154@tcp107 [15057.887185] LNet: Removed LNI 192.168.204.154@tcp108 [15058.911356] LNet: Removed LNI 192.168.204.154@tcp109 [15059.935320] LNet: Removed LNI 192.168.204.154@tcp110 [15059.938639] LNetError: 468165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15059.943708] LNetError: 468165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15059.950674] LNet: Removed LNI 192.168.204.154@tcp111 [15060.959252] LNet: Removed LNI 192.168.204.154@tcp112 [15061.983241] LNet: Removed LNI 192.168.204.154@tcp113 [15063.016107] LNet: Removed LNI 192.168.204.154@tcp114 [15064.031163] LNet: Removed LNI 192.168.204.154@tcp115 [15065.055201] LNet: Removed LNI 192.168.204.154@tcp116 [15066.079188] LNet: Removed LNI 192.168.204.154@tcp117 [15067.104033] LNet: Removed LNI 192.168.204.154@tcp118 [15068.127390] LNet: Removed LNI 192.168.204.154@tcp119 [15068.133219] LNetError: 468165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15068.147538] LNetError: 468165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15068.154443] LNet: Removed LNI 192.168.204.154@tcp120 [15069.215203] LNet: Removed LNI 192.168.204.154@tcp121 [15070.239272] LNet: Removed LNI 192.168.204.154@tcp122 [15070.241468] LNet: Removed LNI 192.168.204.154@tcp123 [15071.263157] LNet: Removed LNI 192.168.204.154@tcp124 [15072.351406] LNet: Removed LNI 192.168.204.154@tcp125 [15072.353718] LNet: Removed LNI 192.168.204.154@tcp126 [15072.366679] LNet: Removed LNI 192.168.204.154@tcp127 [15072.378394] LNet: Removed LNI 192.168.204.154@tcp128 [15072.387787] LNet: Removed LNI 192.168.204.154@tcp129 [15072.400320] LNet: Removed LNI 192.168.204.154@tcp130 [15072.404805] LNet: Removed LNI 192.168.204.154@tcp131 [15072.410840] LNet: Removed LNI 192.168.204.154@tcp132 [15072.417375] LNet: Removed LNI 192.168.204.154@tcp133 [15072.424388] LNet: Removed LNI 192.168.204.154@tcp134 [15072.442595] LNet: Removed LNI 192.168.204.154@tcp135 [15073.530190] Key type .llcrypt unregistered [15073.534716] Key type ._llcrypt unregistered [15099.957956] Key type ._llcrypt registered [15099.960634] Key type .llcrypt registered [15100.051684] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [15114.584267] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [15115.829803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15115.948945] alg: No test for adler32 (adler32-zlib) [15117.039113] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [15117.249963] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [15118.903216] Key type lgssc registered [15119.930501] Lustre: Echo OBD driver; http://www.lustre.org/ [15155.611298] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:05:53 (1788584753) [15177.509881] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [15180.570616] Key type lgssc unregistered [15180.923698] LNet: 472256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15180.929293] LNetError: 472256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15180.964180] LNet: Removed LNI 192.168.204.154@tcp [15181.952574] Key type .llcrypt unregistered [15181.955955] Key type ._llcrypt unregistered [15199.021830] Key type ._llcrypt registered [15199.023203] Key type .llcrypt registered [15199.171859] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [15200.430747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15200.508956] alg: No test for adler32 (adler32-zlib) [15201.626962] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [15202.019468] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [15203.752420] Key type lgssc registered [15205.050339] Lustre: Echo OBD driver; http://www.lustre.org/ [15215.108366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15215.159164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15216.441659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15216.477973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15216.580752] Lustre: lustre-MDT0000: new disk, initializing [15216.649041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15216.665074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15221.637231] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15232.824290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15233.044662] Lustre: 473470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15233.141186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15233.147308] Lustre: Skipped 1 previous similar message [15233.273578] Lustre: lustre-MDT0001: new disk, initializing [15233.425874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15233.483596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15233.511070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15239.390552] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15251.322523] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15257.998605] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15265.321846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15265.636115] Lustre: lustre-OST0000: new disk, initializing [15265.640542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15265.647780] Lustre: 475439:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15265.747938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15271.934276] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15272.008806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15272.018831] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15272.163635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15282.968274] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15288.786254] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing lnet_if_list [15295.241621] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15295.277848] LNet: Added LNI 192.168.204.154@tcp10 [8/256/0/180] [15301.232360] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15301.272029] LNet: Added LNI 192.168.204.154@tcp11 [8/256/0/180] [15306.046272] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15306.108399] LNet: Added LNI 192.168.204.154@tcp12 [8/256/0/180] [15310.200041] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15310.246744] LNet: Added LNI 192.168.204.154@tcp13 [8/256/0/180] [15315.255656] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15315.295227] LNet: Added LNI 192.168.204.154@tcp14 [8/256/0/180] [15319.310726] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15319.361838] LNet: Added LNI 192.168.204.154@tcp15 [8/256/0/180] [15324.465134] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15324.501118] LNet: Added LNI 192.168.204.154@tcp16 [8/256/0/180] [15329.531716] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15329.568909] LNet: Added LNI 192.168.204.154@tcp17 [8/256/0/180] [15334.415421] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15334.444840] LNet: Added LNI 192.168.204.154@tcp18 [8/256/0/180] [15340.089131] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15340.136659] LNet: Added LNI 192.168.204.154@tcp19 [8/256/0/180] [15344.991527] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15345.019677] LNet: Added LNI 192.168.204.154@tcp20 [8/256/0/180] [15350.136329] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15350.165714] LNet: Added LNI 192.168.204.154@tcp21 [8/256/0/180] [15356.077720] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15356.112909] LNet: Added LNI 192.168.204.154@tcp22 [8/256/0/180] [15361.098878] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15361.122899] LNet: Added LNI 192.168.204.154@tcp23 [8/256/0/180] [15367.153630] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15367.189030] LNet: Added LNI 192.168.204.154@tcp24 [8/256/0/180] [15373.823767] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15373.882076] LNet: Added LNI 192.168.204.154@tcp25 [8/256/0/180] [15379.541277] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15379.557712] LNet: Added LNI 192.168.204.154@tcp26 [8/256/0/180] [15385.857937] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15385.928574] LNet: Added LNI 192.168.204.154@tcp27 [8/256/0/180] [15393.150137] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15393.193267] LNet: Added LNI 192.168.204.154@tcp28 [8/256/0/180] [15401.364469] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15401.431179] LNet: Added LNI 192.168.204.154@tcp29 [8/256/0/180] [15409.419468] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15409.456611] LNet: Added LNI 192.168.204.154@tcp30 [8/256/0/180] [15416.103726] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15416.157265] LNet: Added LNI 192.168.204.154@tcp31 [8/256/0/180] [15423.645139] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15423.682363] LNet: Added LNI 192.168.204.154@tcp32 [8/256/0/180] [15430.030645] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15430.071127] LNet: Added LNI 192.168.204.154@tcp33 [8/256/0/180] [15437.367299] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15437.408564] LNet: Added LNI 192.168.204.154@tcp34 [8/256/0/180] [15445.160663] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15445.205985] LNet: Added LNI 192.168.204.154@tcp35 [8/256/0/180] [15453.218412] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15453.256318] LNet: Added LNI 192.168.204.154@tcp36 [8/256/0/180] [15460.986177] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15461.019693] LNet: Added LNI 192.168.204.154@tcp37 [8/256/0/180] [15468.693416] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15468.736453] LNet: Added LNI 192.168.204.154@tcp38 [8/256/0/180] [15476.831154] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15476.881199] LNet: Added LNI 192.168.204.154@tcp39 [8/256/0/180] [15484.561906] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15484.597884] LNet: Added LNI 192.168.204.154@tcp40 [8/256/0/180] [15492.193663] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15492.216064] LNet: Added LNI 192.168.204.154@tcp41 [8/256/0/180] [15499.586950] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15499.608769] LNet: Added LNI 192.168.204.154@tcp42 [8/256/0/180] [15507.121731] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15507.145984] LNet: Added LNI 192.168.204.154@tcp43 [8/256/0/180] [15514.224584] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15514.258365] LNet: Added LNI 192.168.204.154@tcp44 [8/256/0/180] [15521.737754] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15521.781908] LNet: Added LNI 192.168.204.154@tcp45 [8/256/0/180] [15531.931868] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15531.968566] LNet: Added LNI 192.168.204.154@tcp46 [8/256/0/180] [15539.951731] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15539.970559] LNet: Added LNI 192.168.204.154@tcp47 [8/256/0/180] [15547.882108] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15547.913810] LNet: Added LNI 192.168.204.154@tcp48 [8/256/0/180] [15557.940459] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15557.961364] LNet: Added LNI 192.168.204.154@tcp49 [8/256/0/180] [15567.710939] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15567.749876] LNet: Added LNI 192.168.204.154@tcp50 [8/256/0/180] [15591.915358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15591.925539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15591.940976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15594.980109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15594.981908] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15594.986041] Lustre: Skipped 1 previous similar message [15594.991451] Lustre: Skipped 1 previous similar message [15597.400809] Lustre: server umount lustre-MDT0000 complete [15603.320682] LustreError: 473462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585203 with bad export cookie 14141011589371646716 [15603.322615] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15603.351798] LustreError: 473462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15603.774480] Lustre: server umount lustre-MDT0001 complete [15621.344797] Lustre: 472745:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788585205/real 1788585205] req@00000000e0abab0c x1875467096471680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788585221 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15621.386365] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15624.672474] Lustre: 472742:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788585208/real 1788585208] req@00000000ba6b1c93 x1875467096471808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788585224 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15626.783935] Lustre: 472742:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788585210/real 1788585210] req@00000000626f343a x1875467096471936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788585226 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15628.761437] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [15629.795692] Lustre: 472744:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788585213/real 1788585213] req@0000000083fc100c x1875467096472192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788585229 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15634.975344] Lustre: 472744:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788585218/real 1788585218] req@000000004731ad16 x1875467096472320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788585234 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15637.017037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15649.253978] LustreError: 475432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15649.266244] LustreError: 475432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15649.301802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15650.339367] LustreError: 475433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15650.353374] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15652.385250] LustreError: 475432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15655.163157] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15657.442327] LustreError: 475433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15662.563565] LustreError: 475432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15664.847360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15665.686306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15666.771925] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15671.286328] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15680.251123] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15686.261684] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15705.571993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15705.593493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15705.613336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15706.603459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15707.621825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15707.633100] Lustre: Skipped 2 previous similar messages [15711.815500] Lustre: server umount lustre-OST0000 complete [15716.833134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15716.838519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15716.844990] Lustre: Skipped 1 previous similar message [15716.848788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15722.519405] Lustre: server umount lustre-MDT0000 complete [15722.978431] LustreError: 499686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15722.988514] LustreError: 499686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15726.815156] LustreError: 473462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585326 with bad export cookie 14141011589371649383 [15726.827034] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15726.843629] LustreError: 473462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15727.426848] Lustre: server umount lustre-MDT0001 complete [15742.194251] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [15745.261167] Key type lgssc unregistered [15745.648481] LNet: 502480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15745.654645] LNetError: 502480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15745.657950] LNet: Removed LNI 192.168.204.154@tcp [15745.659387] LNet: Removed LNI 192.168.204.154@tcp10 [15745.660960] LNet: Removed LNI 192.168.204.154@tcp11 [15745.662370] LNet: Removed LNI 192.168.204.154@tcp12 [15745.663648] LNet: Removed LNI 192.168.204.154@tcp13 [15745.666115] LNet: Removed LNI 192.168.204.154@tcp14 [15745.672910] LNet: Removed LNI 192.168.204.154@tcp15 [15745.676541] LNet: Removed LNI 192.168.204.154@tcp16 [15745.681354] LNet: Removed LNI 192.168.204.154@tcp17 [15745.688497] LNet: Removed LNI 192.168.204.154@tcp18 [15745.694789] LNet: Removed LNI 192.168.204.154@tcp19 [15745.699522] LNet: Removed LNI 192.168.204.154@tcp20 [15745.703923] LNet: Removed LNI 192.168.204.154@tcp21 [15745.709418] LNet: Removed LNI 192.168.204.154@tcp22 [15745.712929] LNet: Removed LNI 192.168.204.154@tcp23 [15745.715591] LNet: Removed LNI 192.168.204.154@tcp24 [15745.718103] LNet: Removed LNI 192.168.204.154@tcp25 [15745.720892] LNet: Removed LNI 192.168.204.154@tcp26 [15745.724090] LNet: Removed LNI 192.168.204.154@tcp27 [15745.727825] LNet: Removed LNI 192.168.204.154@tcp28 [15745.733978] LNet: Removed LNI 192.168.204.154@tcp29 [15745.737690] LNet: Removed LNI 192.168.204.154@tcp30 [15745.739742] LNet: Removed LNI 192.168.204.154@tcp31 [15745.741946] LNet: Removed LNI 192.168.204.154@tcp32 [15745.745471] LNet: Removed LNI 192.168.204.154@tcp33 [15745.747190] LNet: Removed LNI 192.168.204.154@tcp34 [15745.751669] LNet: Removed LNI 192.168.204.154@tcp35 [15745.754172] LNet: Removed LNI 192.168.204.154@tcp36 [15745.758281] LNet: Removed LNI 192.168.204.154@tcp37 [15745.760283] LNet: Removed LNI 192.168.204.154@tcp38 [15745.762353] LNet: Removed LNI 192.168.204.154@tcp39 [15745.765422] LNet: Removed LNI 192.168.204.154@tcp40 [15745.767171] LNet: Removed LNI 192.168.204.154@tcp41 [15745.774967] LNet: Removed LNI 192.168.204.154@tcp42 [15745.783099] LNet: Removed LNI 192.168.204.154@tcp43 [15745.786252] LNet: Removed LNI 192.168.204.154@tcp44 [15745.788441] LNet: Removed LNI 192.168.204.154@tcp45 [15745.792253] LNet: Removed LNI 192.168.204.154@tcp46 [15745.801119] LNet: Removed LNI 192.168.204.154@tcp47 [15745.808191] LNet: Removed LNI 192.168.204.154@tcp48 [15745.812744] LNet: Removed LNI 192.168.204.154@tcp49 [15745.825886] LNet: Removed LNI 192.168.204.154@tcp50 [15746.376109] Key type .llcrypt unregistered [15746.378574] Key type ._llcrypt unregistered [15768.991687] Key type ._llcrypt registered [15768.995835] Key type .llcrypt registered [15769.159164] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [15784.615802] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [15785.766954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15785.842300] alg: No test for adler32 (adler32-zlib) [15787.139338] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [15787.426093] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [15789.151182] Key type lgssc registered [15790.402797] Lustre: Echo OBD driver; http://www.lustre.org/ [15822.499141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15822.528292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15823.753208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15823.781840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15823.872078] Lustre: lustre-MDT0000: new disk, initializing [15823.924589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15823.938325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15828.060613] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15836.959740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15837.107864] Lustre: 505676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15837.188765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15837.196369] Lustre: Skipped 1 previous similar message [15837.326425] Lustre: lustre-MDT0001: new disk, initializing [15837.449908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15837.486741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15837.499988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15841.706842] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15851.160403] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15856.939420] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15866.099119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15866.415820] Lustre: *** cfs_fail_loc=90f, val=0*** [15866.491045] Lustre: lustre-OST0000: new disk, initializing [15866.498294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15866.508471] Lustre: 507692:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15866.627649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15872.418876] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15873.583267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15873.597923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15873.706759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15881.268819] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15889.376858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15889.386618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15889.393855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15889.398326] Lustre: Skipped 1 previous similar message [15889.404716] Lustre: Skipped 1 previous similar message [15893.520807] Lustre: server umount lustre-OST0000 complete [15899.106817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15899.117347] LustreError: Skipped 1 previous similar message [15899.119141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15899.126713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15902.634012] Lustre: server umount lustre-MDT0000 complete [15906.242437] LustreError: 505669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585506 with bad export cookie 14750095002849840420 [15906.247954] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15906.251928] LustreError: 505669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15906.634357] Lustre: server umount lustre-MDT0001 complete [15919.096904] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [15924.164562] Key type lgssc unregistered [15924.549593] LNet: 509601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15924.564215] LNetError: 509601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15924.577374] LNet: Removed LNI 192.168.204.154@tcp [15925.495299] Key type .llcrypt unregistered [15925.497127] Key type ._llcrypt unregistered [15943.078778] Key type ._llcrypt registered [15943.080377] Key type .llcrypt registered [15943.146126] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [15944.092389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15944.169598] alg: No test for adler32 (adler32-zlib) [15945.158956] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [15945.369787] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [15947.023160] Key type lgssc registered [15948.393813] Lustre: Echo OBD driver; http://www.lustre.org/ [15957.615868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15957.633754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15959.040704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15964.436155] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15968.225617] LustreError: 510872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15968.253029] LustreError: 510872:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15973.349771] LustreError: 510871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15973.391524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15973.739379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15978.614322] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15986.280937] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15992.170294] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15999.103862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15999.464488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16006.021310] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16013.388291] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16026.278513] loop0: detected capacity change from 0 to 204800000 [16026.500287] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16026.515592] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16026.525564] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16032.835348] loop0: detected capacity change from 0 to 204800000 [16033.078296] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16033.089186] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16033.109297] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16038.635476] loop0: detected capacity change from 0 to 204800000 [16044.127860] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16044.295700] Lustre: fs15246-OST0000: new disk, initializing [16044.299919] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16044.307589] Lustre: 514464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16044.366607] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16050.987969] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16060.959264] loop1: detected capacity change from 0 to 204800000 [16067.080128] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16067.205673] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16067.269716] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16067.363967] Lustre: fs15246-MDT0000: new disk, initializing [16067.535389] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16067.585537] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16067.742373] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16068.774476] Lustre: 515587:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000005cfb3501 x1875467876480768/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788585684 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16071.824097] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16077.859110] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16077.866664] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16077.896952] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16097.252565] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16097.264118] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16099.739772] Lustre: server umount fs15246-MDT0000 complete [16102.378986] LustreError: 510868:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16107.496645] LustreError: 512654:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16107.906227] Lustre: server umount fs15246-OST0000 complete [16121.315932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16121.317766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16121.321475] LustreError: Skipped 1 previous similar message [16121.322450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16121.362517] Lustre: Skipped 1 previous similar message [16122.848259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16122.858179] Lustre: Skipped 2 previous similar messages [16125.977872] Lustre: server umount lustre-OST0000 complete [16132.584181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16132.595985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16132.611065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16136.809326] Lustre: server umount lustre-MDT0000 complete [16138.209734] LustreError: 517642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16138.221746] LustreError: 517642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16140.750896] LustreError: 510854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585740 with bad export cookie 4008442589192098110 [16140.758603] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16140.771054] LustreError: 510854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16141.185736] Lustre: server umount lustre-MDT0001 complete [16151.972357] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [16154.719619] Key type lgssc unregistered [16155.047466] LNet: 518687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16155.066241] LNetError: 518687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16155.079905] LNet: Removed LNI 192.168.204.154@tcp [16156.063644] Key type .llcrypt unregistered [16156.068053] Key type ._llcrypt unregistered [16175.526341] Key type ._llcrypt registered [16175.528671] Key type .llcrypt registered [16175.666581] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [16177.036579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16177.063672] alg: No test for adler32 (adler32-zlib) [16178.165700] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [16178.497313] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [16180.215240] Key type lgssc registered [16181.355765] Lustre: Echo OBD driver; http://www.lustre.org/ [16221.707973] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [16231.962284] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [16270.266606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16270.317099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16271.545525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16271.570870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16271.671149] Lustre: lustre-MDT0000: new disk, initializing [16271.767640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16271.798534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16276.917053] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16289.705247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16289.871663] Lustre: 523154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16289.936064] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16289.942256] Lustre: Skipped 1 previous similar message [16290.095495] Lustre: lustre-MDT0001: new disk, initializing [16290.253223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16290.289085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16290.302650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16295.664465] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16307.648884] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16315.191745] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16324.188853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16324.639474] Lustre: lustre-OST0000: new disk, initializing [16324.646671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16324.655874] Lustre: 525123:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16324.772606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16330.356449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16330.368257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16330.513608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16331.497858] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16341.917965] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16347.459647] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16347.770762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16352.669532] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16352.898321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16356.322732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16356.329101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16356.333562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16356.356814] Lustre: Skipped 1 previous similar message [16361.595540] Lustre: server umount lustre-OST0000 complete [16366.563307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16366.569152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16366.585654] Lustre: Skipped 1 previous similar message [16366.619436] Lustre: Skipped 2 previous similar messages [16371.680789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16371.695046] Lustre: Skipped 1 previous similar message [16372.206535] Lustre: server umount lustre-MDT0000 complete [16376.635104] LustreError: 523949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585976 with bad export cookie 12405564596345853446 [16376.635947] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16376.648610] LustreError: 523949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16377.022285] Lustre: server umount lustre-MDT0001 complete [16404.714624] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [16418.038952] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [16454.253145] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16161 sec ========= 01:27:32 (1788586052) [16456.056739] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:27:34 (1788586054) === [16460.039709] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:27:38 (1788586058) === [16501.778545] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [16505.143853] Key type lgssc unregistered [16505.409040] LNet: 531773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16505.431131] LNetError: 531773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16505.446718] LNet: Removed LNI 192.168.204.154@tcp [16506.753982] Key type .llcrypt unregistered [16506.756048] Key type ._llcrypt unregistered