[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480137879 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002392] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009146] LSM: Security Framework initializing [ 0.010047] Yama: becoming mindful. [ 0.011038] SELinux: Initializing. [ 0.012074] *** VALIDATE selinux *** [ 0.020616] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025498] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028119] *** VALIDATE tmpfs *** [ 0.030165] *** VALIDATE proc *** [ 0.032133] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.035167] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037072] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.042573] debug: unmapping init [mem 0xffffffff90c59000-0xffffffff90c60fff] [ 0.045155] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046690] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054320] rcu: Hierarchical SRCU implementation. [ 0.056404] smp: Bringing up secondary CPUs ... [ 0.057597] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.061434] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146603] node 0 deferred pages initialised in 79ms [ 0.148154] devtmpfs: initialized [ 0.150251] x86/mm: Memory block size: 128MB [ 0.152816] gcov: version magic: 0x41383552 [ 0.156287] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157133] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158384] pinctrl core: initialized pinctrl subsystem [ 0.159202] [ 0.159989] ************************************************************* [ 0.160016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161014] ** ** [ 0.162044] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163014] ** ** [ 0.164018] ** This means that this kernel is built to expose internal ** [ 0.165015] ** IOMMU data structures, which may compromise security on ** [ 0.166019] ** your system. ** [ 0.167016] ** ** [ 0.168013] ** If you see this message and you are not debugging the ** [ 0.169013] ** kernel, report this immediately to your vendor! ** [ 0.170017] ** ** [ 0.171015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172020] ************************************************************* [ 0.173961] NET: Registered protocol family 16 [ 0.174454] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177645] cpuidle: using governor menu [ 0.178000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182484] PCI: Using configuration type 1 for base access [ 0.185144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195142] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198035] cryptd: max_cpu_qlen set to 1000 [ 0.200250] ACPI: Added _OSI(Module Device) [ 0.201016] ACPI: Added _OSI(Processor Device) [ 0.202015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.203015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207793] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209415] ACPI: Interpreter enabled [ 0.211062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.213014] ACPI: Using IOAPIC for interrupt routing [ 0.214109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.218401] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.228307] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.234022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.238137] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.243478] acpiphp: Slot [2] registered [ 0.246126] acpiphp: Slot [5] registered [ 0.247144] acpiphp: Slot [6] registered [ 0.249135] acpiphp: Slot [7] registered [ 0.251131] acpiphp: Slot [8] registered [ 0.252118] acpiphp: Slot [9] registered [ 0.254133] acpiphp: Slot [10] registered [ 0.256118] acpiphp: Slot [3] registered [ 0.257104] acpiphp: Slot [4] registered [ 0.259140] acpiphp: Slot [11] registered [ 0.260090] acpiphp: Slot [12] registered [ 0.262113] acpiphp: Slot [13] registered [ 0.263178] acpiphp: Slot [14] registered [ 0.265098] acpiphp: Slot [15] registered [ 0.267137] acpiphp: Slot [16] registered [ 0.268148] acpiphp: Slot [17] registered [ 0.270178] acpiphp: Slot [18] registered [ 0.272116] acpiphp: Slot [19] registered [ 0.273131] acpiphp: Slot [20] registered [ 0.275104] acpiphp: Slot [21] registered [ 0.276138] acpiphp: Slot [22] registered [ 0.279271] acpiphp: Slot [23] registered [ 0.281122] acpiphp: Slot [24] registered [ 0.282131] acpiphp: Slot [25] registered [ 0.284105] acpiphp: Slot [26] registered [ 0.286117] acpiphp: Slot [27] registered [ 0.288122] acpiphp: Slot [28] registered [ 0.289145] acpiphp: Slot [29] registered [ 0.291129] acpiphp: Slot [30] registered [ 0.293145] acpiphp: Slot [31] registered [ 0.295074] PCI host bridge to bus 0000:00 [ 0.297022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.299021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.301033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.304030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.307027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.309020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.311159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.314212] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.318498] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.328995] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.335691] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.338026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.341024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.344024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.348467] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.351885] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.354043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.357938] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.363017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.379018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.384021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.392269] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.399018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.408016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.426022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.437767] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.455016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.473027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.483154] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.490014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.500018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.516022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.526119] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.532020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.540017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.566017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.579000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.583919] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.592016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.610000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.624447] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.638017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.647016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.671019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.686000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.689393] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.693415] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.695267] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.697279] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.702159] iommu: Default domain type: Passthrough [ 0.704420] SCSI subsystem initialized [ 0.706133] ACPI: bus type USB registered [ 0.707127] usbcore: registered new interface driver usbfs [ 0.710095] usbcore: registered new interface driver hub [ 0.712094] usbcore: registered new device driver usb [ 0.714239] pps_core: LinuxPPS API ver. 1 registered [ 0.715013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.718064] PTP clock support registered [ 0.721111] EDAC MC: Ver: 3.0.0 [ 0.722501] PCI: Using ACPI for IRQ routing [ 0.725567] NetLabel: Initializing [ 0.727014] NetLabel: domain hash size = 128 [ 0.729016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.732101] NetLabel: unlabeled traffic allowed by default [ 0.734162] vgaarb: loaded [ 0.736215] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.739018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.745643] clocksource: Switched to clocksource kvm-clock [ 0.856600] VFS: Disk quotas dquot_6.6.0 [ 0.858188] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.861077] *** VALIDATE ramfs *** [ 0.862228] *** VALIDATE hugetlbfs *** [ 0.863673] pnp: PnP ACPI init [ 0.865855] pnp: PnP ACPI: found 6 devices [ 0.882832] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.885554] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.887978] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.889683] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.892303] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.894490] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.897182] NET: Registered protocol family 2 [ 0.899739] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.904630] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.908484] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.912972] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.916665] TCP: Hash tables configured (established 65536 bind 65536) [ 0.919835] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.923104] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.926392] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.929962] NET: Registered protocol family 1 [ 0.932784] RPC: Registered named UNIX socket transport module. [ 0.935953] RPC: Registered udp transport module. [ 0.937748] RPC: Registered tcp transport module. [ 0.939641] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.942370] NET: Registered protocol family 44 [ 0.944245] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.946371] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.948623] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.951791] PCI: CLS 0 bytes, default 64 [ 0.954863] Unpacking initramfs... [ 2.418273] debug: unmapping init [mem 0xffff88bc3cc54000-0xffff88bc3ffbffff] [ 2.424737] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.426865] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.430715] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.920127] Initialise system trusted keyrings [ 2.922407] Key type blacklist registered [ 2.924463] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.933442] zbud: loaded [ 2.936581] *** VALIDATE nfs *** [ 2.939498] *** VALIDATE nfs4 *** [ 2.940956] pstore: using deflate compression [ 2.944181] Platform Keyring initialized [ 3.051816] NET: Registered protocol family 38 [ 3.054375] Key type asymmetric registered [ 3.056412] Asymmetric key parser 'x509' registered [ 3.058809] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.062662] io scheduler mq-deadline registered [ 3.064200] io scheduler kyber registered [ 3.065641] io scheduler bfq registered [ 3.067490] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.070290] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.072956] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.075554] ACPI: Power Button [PWRF] [ 3.081151] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.088213] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.106188] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.118109] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.149639] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.177529] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.205627] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.210325] Non-volatile memory driver v1.3 [ 3.212175] Linux agpgart interface v0.103 [ 3.243755] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.246832] vda: detected capacity change from 0 to 74645504 [ 3.266024] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.269590] vdb: detected capacity change from 0 to 1073741824 [ 3.287167] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.289746] vdc: detected capacity change from 0 to 2621440000 [ 3.306341] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.310692] vdd: detected capacity change from 0 to 2621440000 [ 3.326048] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.328740] vde: detected capacity change from 0 to 4294967296 [ 3.344549] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.347711] vdf: detected capacity change from 0 to 4294967296 [ 3.355941] libphy: Fixed MDIO Bus: probed [ 3.362508] usbcore: registered new interface driver usbserial_generic [ 3.365692] usbserial: USB Serial support registered for generic [ 3.368753] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.373408] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.375604] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.378455] mousedev: PS/2 mouse device common for all mice [ 3.382528] rtc_cmos 00:05: RTC can wake from S4 [ 3.385345] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.389684] rtc_cmos 00:05: registered as rtc0 [ 3.391602] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.392758] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.394595] intel_pstate: CPU model not supported [ 3.400751] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.402075] hid: raw HID events driver (C) Jiri Kosina [ 3.407518] usbcore: registered new interface driver usbhid [ 3.409294] usbhid: USB HID core driver [ 3.410678] drop_monitor: Initializing network drop monitor service [ 3.412840] Initializing XFRM netlink socket [ 3.414775] NET: Registered protocol family 10 [ 3.417908] Segment Routing with IPv6 [ 3.419294] NET: Registered protocol family 17 [ 3.421366] mpls_gso: MPLS GSO support [ 3.427647] RAS: Correctable Errors collector initialized. [ 3.429537] AVX version of gcm_enc/dec engaged. [ 3.431102] AES CTR mode by8 optimization enabled [ 3.512337] sched_clock: Marking stable (3512303281, 0)->(4500045687, -987742406) [ 3.515706] registered taskstats version 1 [ 3.517587] Loading compiled-in X.509 certificates [ 3.519678] zswap: loaded using pool lzo/zbud [ 3.545730] Key type big_key registered [ 3.558505] Key type encrypted registered [ 3.560169] ima: No TPM chip found, activating TPM-bypass! [ 3.562576] ima: Allocated hash algorithm: sha1 [ 3.564093] ima: No architecture policies found [ 3.565732] evm: Initialising EVM extended attributes: [ 3.567776] evm: security.selinux [ 3.569288] evm: security.ima [ 3.570310] evm: security.capability [ 3.571590] evm: HMAC attrs: 0x1 [ 3.573962] rtc_cmos 00:05: setting system clock to 2026-07-13 18:46:24 UTC (1783968384) [ 3.580265] debug: unmapping init [mem 0xffffffff91c03000-0xffffffff91dfffff] [ 3.583160] debug: unmapping init [mem 0xffffffff90982000-0xffffffff90c58fff] [ 3.589106] Write protecting the kernel read-only data: 28672k [ 3.593285] debug: unmapping init [mem 0xffffffff8f003000-0xffffffff8f1fffff] [ 3.596604] debug: unmapping init [mem 0xffffffff8f914000-0xffffffff8f9fffff] [ 3.634277] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.643666] systemd[1]: Detected virtualization kvm. [ 3.645800] systemd[1]: Detected architecture x86-64. [ 3.647891] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.675408] systemd[1]: No hostname configured. [ 3.677635] systemd[1]: Set hostname to . [ 3.680313] random: systemd: uninitialized urandom read (16 bytes read) [ 3.683271] systemd[1]: Initializing machine ID from random generator. [ 3.829191] random: systemd: uninitialized urandom read (16 bytes read) [ 3.832334] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.837094] random: systemd: uninitialized urandom read (16 bytes read) [ 3.840082] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.844728] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.501898] device-mapper: uevent: version 1.0.3 [ 4.503791] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.167276] virtio_net virtio0 ens2: renamed from eth0 [ 5.179522] random: fast init done [ 5.352071] scsi host0: ata_piix [ 5.417802] scsi host1: ata_piix [ 5.419195] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.421812] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.918214] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.077119] random: crng init done [ 10.078779] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.466095] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.579361] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.853418] SELinux: Disabled at runtime. [ 11.909461] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.918932] systemd[1]: Detected virtualization kvm. [ 11.920597] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.432860] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.436194] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.441783] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.445551] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.448878] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.455055] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.459637] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ 12.614393] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.920522] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.229953] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.239857] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.348903] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.366047] EDAC sbridge: Ver: 1.1.2 [ 15.013160] Key type dns_resolver registered [ 15.311201] NFS: Registering the id_resolver key type [ 15.313361] Key type id_resolver registered [ 15.314887] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg416-server login: [ 40.716097] hrtimer: interrupt took 8088806 ns [ 47.905516] libcfs: loading out-of-tree module taints kernel. [ 48.011419] Key type ._llcrypt registered [ 48.016715] Key type .llcrypt registered [ 48.120404] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 70.623334] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 72.112110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 72.130959] alg: No test for adler32 (adler32-zlib) [ 73.760494] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 74.744921] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 76.527191] Key type lgssc registered [ 78.138153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.227706] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 144.132773] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 158.804883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 158.840396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.140894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 160.186618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 160.319350] Lustre: lustre-MDT0000: new disk, initializing [ 160.452687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.494460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.470700] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.241086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.371353] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.406825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.411588] Lustre: Skipped 1 previous similar message [ 179.514137] Lustre: lustre-MDT0001: new disk, initializing [ 179.610638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.651173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.682427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.226593] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.313234] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.005809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.234800] Lustre: lustre-OST0000: new disk, initializing [ 201.240462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 201.248218] Lustre: 8445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.341145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.317472] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.481345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 210.500207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 210.569579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 224.728522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.845221] Lustre: lustre-OST0001: new disk, initializing [ 224.848691] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 224.853995] Lustre: 9518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.923790] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.207544] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.465314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 232.478223] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 232.565426] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.251424] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 254.642640] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 261.002579] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.251010] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing yml_node [ 270.234946] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 272.807881] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 275.411280] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 276.874889] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 13 14:50:55 EDT 2026 [ 292.990533] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 294.513215] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.799980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 312.807518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.816458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.336987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.337507] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.348421] Lustre: Skipped 2 previous similar messages [ 314.369918] Lustre: Skipped 2 previous similar messages [ 315.947714] Lustre: server umount lustre-MDT0000 complete [ 319.775938] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783968700 with bad export cookie 7411282525406666663 [ 319.791132] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 319.798769] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 324.579692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.580253] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.582169] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.597517] Lustre: Skipped 1 previous similar message [ 324.632236] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 329.702136] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.702548] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.741284] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 329.755974] Lustre: Skipped 2 previous similar messages [ 330.720859] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.736109] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 334.303129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 334.520391] Lustre: server umount lustre-MDT0001 complete [ 338.777294] Lustre: server umount lustre-OST0000 complete [ 342.808805] Lustre: server umount lustre-OST0001 complete [ 349.314815] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 359.110389] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 395.977873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.249521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 396.315390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 396.408829] Lustre: lustre-MDT0000: new disk, initializing [ 396.515749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 396.532818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.103377] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 412.747907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.910325] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 413.017252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 413.023833] Lustre: Skipped 1 previous similar message [ 413.171744] Lustre: lustre-MDT0001: new disk, initializing [ 413.301135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 413.356375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 413.372501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 418.756556] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 429.360309] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 435.321953] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 441.620570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 441.829863] Lustre: lustre-OST0000: new disk, initializing [ 441.835856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 441.840808] Lustre: 17046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 441.905460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.766953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 443.782676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 443.910522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 448.111967] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 456.668638] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 461.385774] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 461.605663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 465.950719] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 466.179101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 469.480970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 469.482773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 469.492369] LustreError: Skipped 1 previous similar message [ 469.493219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 469.502024] Lustre: Skipped 1 previous similar message [ 469.527044] Lustre: Skipped 3 previous similar messages [ 474.606622] LustreError: 17031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.642943] LustreError: 17031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 474.784066] Lustre: server umount lustre-OST0000 complete [ 479.714329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.718321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 479.726973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 479.737438] Lustre: Skipped 1 previous similar message [ 485.336075] Lustre: server umount lustre-MDT0000 complete [ 489.980295] LustreError: 15082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.040508] LustreError: 15082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 490.339569] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783968871 with bad export cookie 7411282525406668196 [ 490.345589] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 490.350568] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 490.901863] Lustre: server umount lustre-MDT0001 complete [ 495.978757] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 14:54:34 (1783968874) [ 503.954450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.718663] LustreError: 19309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.801620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.939402] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 517.374153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.137310] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 530.150285] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 536.912337] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 545.002526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 552.579852] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 561.071631] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 572.901206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 572.914104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.928563] Lustre: Skipped 1 previous similar message [ 572.933635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 572.944193] Lustre: Skipped 2 previous similar messages [ 578.246088] Lustre: server umount lustre-OST0000 complete [ 587.890893] Lustre: server umount lustre-MDT0000 complete [ 588.769267] LustreError: 19305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 588.791079] LustreError: 19305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 591.339333] LustreError: 22091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783968972 with bad export cookie 7411282525406669071 [ 591.346380] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.354705] LustreError: 22091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 591.661948] Lustre: server umount lustre-MDT0001 complete [ 600.775375] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 603.777290] Key type lgssc unregistered [ 604.064279] LNet: 22842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 604.074842] LNetError: 22842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 604.087688] LNet: Removed LNI 192.168.204.116@tcp [ 605.096199] Key type .llcrypt unregistered [ 605.101907] Key type ._llcrypt unregistered [ 625.984827] Key type ._llcrypt registered [ 625.989918] Key type .llcrypt registered [ 626.117570] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 627.312490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 627.342810] alg: No test for adler32 (adler32-zlib) [ 628.410575] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 628.646344] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 630.304093] Key type lgssc registered [ 631.558056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 639.397359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 639.410265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.920956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 645.285112] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 650.211526] LustreError: 24233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.229188] LustreError: 24233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 653.210492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.559679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 657.669745] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 665.436699] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 670.402936] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 677.045250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 677.356121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 683.583224] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 686.588901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 690.676824] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 706.318549] Lustre: server umount lustre-OST0000 complete [ 710.114189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 710.125076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 710.154794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.168871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.180834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 717.283492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 717.289425] Lustre: Skipped 2 previous similar messages [ 719.396865] Lustre: server umount lustre-MDT0000 complete [ 722.404020] LustreError: 25478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.417454] LustreError: 25478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 722.902711] LustreError: 24215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969103 with bad export cookie 12734157120937509959 [ 722.910448] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.917041] LustreError: 24215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 732.332743] Lustre: server umount lustre-MDT0001 complete [ 742.686735] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 745.646340] Key type lgssc unregistered [ 746.049823] LNet: 28215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.063477] LNetError: 28215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.099569] LNet: Removed LNI 192.168.204.116@tcp [ 747.101857] Key type .llcrypt unregistered [ 747.103624] Key type ._llcrypt unregistered [ 772.151450] Key type ._llcrypt registered [ 772.162255] Key type .llcrypt registered [ 772.337135] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 773.014154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 773.107262] alg: No test for adler32 (adler32-zlib) [ 774.187580] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 774.463423] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 776.209267] Key type lgssc registered [ 778.222149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 789.850635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 789.894065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.879957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 797.043485] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 801.253518] LustreError: 29630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 801.260032] LustreError: 29630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 805.304740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.589615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 809.570275] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 817.708861] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 823.524604] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 836.582103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 836.836378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.829961] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 850.367130] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 861.442459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 870.881096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 870.893588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.938144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 871.907797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 871.908433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.971786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 873.524905] Lustre: server umount lustre-OST0000 complete [ 877.535951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 877.547309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.572153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 877.576303] Lustre: Skipped 1 previous similar message [ 882.144465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 882.147452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.648196] Lustre: server umount lustre-MDT0000 complete [ 887.270187] LustreError: 29630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.377441] LustreError: 29611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969268 with bad export cookie 10918952506458754867 [ 887.384147] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 887.392602] LustreError: 29611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 887.732630] Lustre: server umount lustre-MDT0001 complete [ 902.373895] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 906.574286] Key type lgssc unregistered [ 907.138978] LNet: 33547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.153520] LNetError: 33547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.211502] LNet: Removed LNI 192.168.204.116@tcp [ 909.369435] Key type .llcrypt unregistered [ 909.378197] Key type ._llcrypt unregistered [ 938.333809] Key type ._llcrypt registered [ 938.340329] Key type .llcrypt registered [ 938.482217] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 939.168473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 939.293563] alg: No test for adler32 (adler32-zlib) [ 940.458599] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 940.797668] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 942.471191] Key type lgssc registered [ 943.606227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 951.151932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 951.189674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.658416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 956.708727] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 963.049770] LustreError: 34970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 963.061440] LustreError: 34970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 965.048305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 965.418266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 969.649354] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 977.258860] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 984.102341] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 991.645246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 992.047023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.185893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 998.541219] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1005.399342] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1017.829567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1017.850120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.871154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.916757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1020.929650] LustreError: Skipped 1 previous similar message [ 1020.937246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.937990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.972695] Lustre: Skipped 1 previous similar message [ 1023.375526] Lustre: server umount lustre-OST0000 complete [ 1031.145406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.146861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1031.154638] Lustre: Skipped 1 previous similar message [ 1031.164310] Lustre: Skipped 1 previous similar message [ 1033.462868] Lustre: server umount lustre-MDT0000 complete [ 1036.264301] LustreError: 35700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1036.287583] LustreError: 35700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1036.877884] LustreError: 34952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969417 with bad export cookie 13842151745653344033 [ 1036.884036] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1036.887765] LustreError: 34952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1037.141785] Lustre: server umount lustre-MDT0001 complete [ 1048.271297] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1051.073897] Key type lgssc unregistered [ 1051.378812] LNet: 38508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.387638] LNetError: 38508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.407377] LNet: Removed LNI 192.168.204.116@tcp [ 1052.287135] Key type .llcrypt unregistered [ 1052.293294] Key type ._llcrypt unregistered [ 1075.457498] Key type ._llcrypt registered [ 1075.461600] Key type .llcrypt registered [ 1075.599157] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1076.680082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1076.795384] alg: No test for adler32 (adler32-zlib) [ 1077.918952] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 1078.154851] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1079.919141] Key type lgssc registered [ 1081.131891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1089.291715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1089.315653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.786372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.964219] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1100.258782] LustreError: 39923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1100.296108] LustreError: 39923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1103.479930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.834260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1108.143245] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1115.142121] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1120.067101] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1125.933484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1126.248566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.321112] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1135.468607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1139.713681] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1147.714978] Lustre: lustre-OST0000: Not available for connect from 192.168.204.16@tcp (stopping) [ 1149.929688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1149.942063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.964509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1150.944952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1150.961048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.978091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1150.987094] Lustre: Skipped 1 previous similar message [ 1153.018206] Lustre: server umount lustre-OST0000 complete [ 1175.527609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1175.529680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.543969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1175.549544] Lustre: Skipped 2 previous similar messages [ 1180.083900] Lustre: server umount lustre-MDT0000 complete [ 1181.665607] LustreError: 40651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1181.689044] LustreError: 40651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1184.330977] LustreError: 41767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969565 with bad export cookie 14573858204598476634 [ 1184.338175] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1184.351917] LustreError: 41767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1190.956410] Lustre: server umount lustre-MDT0001 complete [ 1202.446918] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1205.241800] Key type lgssc unregistered [ 1205.519712] LNet: 43631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.525908] LNetError: 43631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1205.543819] LNet: Removed LNI 192.168.204.116@tcp [ 1206.352053] Key type .llcrypt unregistered [ 1206.353912] Key type ._llcrypt unregistered [ 1230.418589] Key type ._llcrypt registered [ 1230.422421] Key type .llcrypt registered [ 1230.546723] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1231.507160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1231.526678] alg: No test for adler32 (adler32-zlib) [ 1232.696722] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 1233.015736] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1234.695584] Key type lgssc registered [ 1236.038749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1244.524620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1244.572654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.020507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1250.610168] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1255.406888] LustreError: 45043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1255.420917] LustreError: 45043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1259.552155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.168880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1265.426740] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1274.451298] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1281.312181] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1289.479263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1290.086243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.126954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1296.909168] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1306.309547] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1313.652827] Lustre: server umount lustre-MDT0000 complete [ 1314.596527] LustreError: 45778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.790936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1314.829828] Lustre: Skipped 2 previous similar messages [ 1317.794875] LustreError: 45023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969698 with bad export cookie 2752365116127100747 [ 1317.800968] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1317.805727] LustreError: 45023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1319.661837] LustreError: 45039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.691456] LustreError: 45039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1319.733656] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.16@tcp (stopping) [ 1319.916090] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.912705] LustreError: 45778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1322.923415] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1322.934744] LustreError: 45778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1322.946718] Lustre: Skipped 1 previous similar message [ 1324.787134] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.16@tcp (stopping) [ 1328.033514] LustreError: 46951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.041278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1328.059145] LustreError: 46951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1332.196452] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1332.464887] Lustre: server umount lustre-MDT0001 complete [ 1362.399138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1362.566962] Lustre: server umount lustre-OST0000 complete [ 1381.640384] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1384.689354] Key type lgssc unregistered [ 1385.063627] LNet: 48908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.072022] LNetError: 48908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.088896] LNet: Removed LNI 192.168.204.116@tcp [ 1386.129391] Key type .llcrypt unregistered [ 1386.133685] Key type ._llcrypt unregistered [ 1411.549228] Key type ._llcrypt registered [ 1411.552767] Key type .llcrypt registered [ 1411.719838] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1412.662791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1412.746039] alg: No test for adler32 (adler32-zlib) [ 1413.877985] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 1414.137880] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1415.831193] Key type lgssc registered [ 1417.257848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1425.346460] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1425.372152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1440.927424] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1446.047973] LustreError: 50297:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 1472.801879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1480.708799] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1489.841972] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1654.992979] Lustre: server umount lustre-OST0000 complete [ 1673.343749] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1676.509423] Key type lgssc unregistered [ 1676.872450] LNet: 51973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.885086] LNetError: 51973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.921851] LNet: Removed LNI 192.168.204.116@tcp [ 1677.916564] Key type .llcrypt unregistered [ 1677.919133] Key type ._llcrypt unregistered [ 1705.144784] Key type ._llcrypt registered [ 1705.150209] Key type .llcrypt registered [ 1705.268306] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1706.692850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1706.756595] alg: No test for adler32 (adler32-zlib) [ 1707.868563] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 1708.163104] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1709.951181] Key type lgssc registered [ 1711.413934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1720.052531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1720.079308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.737429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.586993] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1732.064839] LustreError: 53393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1732.074700] LustreError: 53393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1735.092719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1735.370824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1741.180490] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1751.290158] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1758.461748] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1765.028430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1765.445753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1766.524341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1772.315492] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1780.815948] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1786.342298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.349392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1786.349647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1786.355081] Lustre: Skipped 1 previous similar message [ 1789.835499] Lustre: server umount lustre-OST0000 complete [ 1796.583788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.610534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1796.616337] Lustre: Skipped 2 previous similar messages [ 1799.820110] Lustre: server umount lustre-MDT0000 complete [ 1801.701959] LustreError: 53393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1801.733476] LustreError: 53393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1803.987641] LustreError: 53372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970184 with bad export cookie 3103126255097785922 [ 1804.000846] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1804.009583] LustreError: 53372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1804.306532] Lustre: server umount lustre-MDT0001 complete [ 1817.662085] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1820.786421] Key type lgssc unregistered [ 1821.264749] LNet: 56924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.278899] LNetError: 56924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.296829] LNet: Removed LNI 192.168.204.116@tcp [ 1822.839815] Key type .llcrypt unregistered [ 1822.843657] Key type ._llcrypt unregistered [ 1851.716413] Key type ._llcrypt registered [ 1851.718438] Key type .llcrypt registered [ 1851.904551] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1852.920653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1852.937958] alg: No test for adler32 (adler32-zlib) [ 1854.119437] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 1854.446984] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1856.178084] Key type lgssc registered [ 1857.427545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1865.602625] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1865.623833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1881.247437] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1886.367518] LustreError: 58314:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 1913.057416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1919.640674] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1927.202678] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1934.262629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1934.506667] LustreError: 57650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000041895039 x1870628373137664/t0(0) o250->MGC192.168.204.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1934.809336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1934.901519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1939.944560] LustreError: 58339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1939.976480] LustreError: 58339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1940.069175] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1945.064535] LustreError: 58340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.110693] LustreError: 59348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.130556] LustreError: 59348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1948.567675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.976025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1953.989123] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1963.183711] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1970.042800] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1974.768818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.791296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1974.792315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1978.394799] Lustre: server umount lustre-OST0000 complete [ 2004.451602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.463584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2004.471972] Lustre: Skipped 2 previous similar messages [ 2004.483292] Lustre: Skipped 2 previous similar messages [ 2007.389898] Lustre: server umount lustre-MDT0000 complete [ 2009.578474] LustreError: 59348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.601912] LustreError: 59348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2011.223463] LustreError: 61053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970392 with bad export cookie 17633951682942608917 [ 2011.225255] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2011.246226] LustreError: 61053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2017.657548] Lustre: server umount lustre-MDT0001 complete [ 2029.719113] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2033.450250] Key type lgssc unregistered [ 2033.803558] LNet: 62059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.817595] LNetError: 62059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.848249] LNet: Removed LNI 192.168.204.116@tcp [ 2034.744215] Key type .llcrypt unregistered [ 2034.746572] Key type ._llcrypt unregistered [ 2060.536404] Key type ._llcrypt registered [ 2060.541464] Key type .llcrypt registered [ 2060.682182] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2062.048374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.094416] alg: No test for adler32 (adler32-zlib) [ 2063.155668] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 2063.390451] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2065.079319] Key type lgssc registered [ 2066.465328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2077.279132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2077.306106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.807435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.117343] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2088.930797] LustreError: 63472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.944696] LustreError: 63472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2094.055331] LustreError: 63473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2095.090432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.762679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2100.837863] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2108.353574] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2113.768290] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2121.053860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2121.392181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2123.440511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2127.475166] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2135.439963] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2154.466402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2154.478343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.518358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2156.526276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2156.527435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2156.536918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2161.637592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2161.646719] Lustre: Skipped 2 previous similar messages [ 2166.765073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2166.770203] Lustre: Skipped 1 previous similar message [ 2168.287150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2168.519465] Lustre: server umount lustre-OST0000 complete [ 2177.001813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.006223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2177.014763] Lustre: Skipped 1 previous similar message [ 2180.821663] Lustre: server umount lustre-MDT0000 complete [ 2182.113253] LustreError: 63472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.140214] LustreError: 63472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2185.450084] LustreError: 63454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970566 with bad export cookie 6839054902241660256 [ 2185.453633] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2185.463163] LustreError: 63454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2185.947287] Lustre: server umount lustre-MDT0001 complete [ 2199.401821] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2202.640843] Key type lgssc unregistered [ 2202.943139] LNet: 67044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.952779] LNetError: 67044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2202.969364] LNet: Removed LNI 192.168.204.116@tcp [ 2204.085454] Key type .llcrypt unregistered [ 2204.093185] Key type ._llcrypt unregistered [ 2234.480404] Key type ._llcrypt registered [ 2234.483415] Key type .llcrypt registered [ 2234.593502] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2235.618783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2235.748457] alg: No test for adler32 (adler32-zlib) [ 2236.771186] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 2237.073485] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2238.743169] Key type lgssc registered [ 2239.927757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2249.741571] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 15:23:48 (1783970628) [ 2256.942704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2256.964162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.467510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2263.444543] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2268.649489] LustreError: 68749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.662557] LustreError: 68749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2272.942061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.513147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2278.600766] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2287.050701] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2294.270817] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2301.367444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2301.750664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.798201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2308.295786] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2317.186832] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2324.951191] Lustre: Failing over lustre-MDT0000 [ 2324.969992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -19 [ 2324.983163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.004417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2325.272259] LustreError: 69961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.530631] Lustre: server umount lustre-MDT0000 complete [ 2328.552454] LustreError: 68749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.557944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.583315] LustreError: 68749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2328.599388] Lustre: Skipped 1 previous similar message [ 2333.676482] LustreError: 69476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.704477] LustreError: 69476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2337.866802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.025770] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2338.278296] Lustre: *** cfs_fail_loc=135, val=0*** [ 2338.509860] LustreError: 71834:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2338.535717] LustreError: 71834:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.116@tcp: cfg command failed: rc = -2 [ 2338.555774] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2338.555774] [ 2338.577512] LustreError: MGC192.168.204.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2338.608567] LustreError: 71823:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2338.627582] LustreError: 71823:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2338.641863] LustreError: 71823:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2338.651254] LustreError: 71823:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2338.679869] Lustre: server umount lustre-MDT0000 complete [ 2338.687687] LustreError: 71823:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2338.789505] LustreError: 68744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.820420] LustreError: 68744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2346.835965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.957919] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2346.973926] LustreError: 68740:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2347.246384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.323957] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2350.828977] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2352.637577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2352.683860] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2352.745426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2352.850174] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2362.859921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.878895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2362.887310] Lustre: Skipped 1 previous similar message [ 2367.981727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2367.989309] Lustre: Skipped 1 previous similar message [ 2369.118220] Lustre: server umount lustre-OST0000 complete [ 2378.216568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.220866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2378.235712] Lustre: Skipped 1 previous similar message [ 2378.237994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.264144] Lustre: Skipped 1 previous similar message [ 2379.565891] Lustre: server umount lustre-MDT0000 complete [ 2383.329530] LustreError: 69476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.353478] LustreError: 69476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2383.782973] LustreError: 68729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970764 with bad export cookie 18003964246661059264 [ 2383.793226] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.803809] LustreError: 68729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2384.303321] Lustre: server umount lustre-MDT0001 complete [ 2396.068643] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2399.301845] Key type lgssc unregistered [ 2399.573409] LNet: 73619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.582386] LNetError: 73619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2399.603774] LNet: Removed LNI 192.168.204.116@tcp [ 2400.604231] Key type .llcrypt unregistered [ 2400.606462] Key type ._llcrypt unregistered [ 2427.734795] Key type ._llcrypt registered [ 2427.738542] Key type .llcrypt registered [ 2427.862419] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2429.063390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2429.107884] alg: No test for adler32 (adler32-zlib) [ 2430.404897] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 2430.873671] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2432.687234] Key type lgssc registered [ 2434.038199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2442.631397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2442.654835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.099402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2448.870319] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2453.473411] LustreError: 75032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.492227] LustreError: 75032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2458.051310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.506303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2463.097524] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2470.880814] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2477.912900] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2485.435492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2485.773623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.881828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2492.279699] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2501.411663] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2507.667534] Lustre: Failing over lustre-MDT0000 [ 2508.069458] Lustre: server umount lustre-MDT0000 complete [ 2508.275840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.283277] LustreError: 76873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.298812] Lustre: Skipped 1 previous similar message [ 2513.397990] LustreError: 76873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.432368] LustreError: 76873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2518.502461] LustreError: 76873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.524566] LustreError: 76873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2519.260253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.400697] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2519.611438] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2519.787629] LustreError: 78121:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2519.807059] LustreError: 78121:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.116@tcp: cfg command failed: rc = -9 [ 2519.819320] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2519.819320] [ 2519.830173] LustreError: MGC192.168.204.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2519.843520] LustreError: 78109:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2519.851704] LustreError: 78109:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2519.867921] LustreError: 78109:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2519.874029] LustreError: 78109:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2519.915174] Lustre: server umount lustre-MDT0000 complete [ 2519.918475] LustreError: 78109:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2523.621274] LustreError: 75032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2523.644744] LustreError: 75032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2527.586579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.777080] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2527.789809] LustreError: 75024:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2528.124189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.177954] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2529.005254] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2533.307856] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2533.362202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2533.456187] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2533.517701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2543.587883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.592265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2543.628511] Lustre: Skipped 2 previous similar messages [ 2548.524770] Lustre: server umount lustre-OST0000 complete [ 2553.825642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2553.831061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.844376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.844384] Lustre: Skipped 1 previous similar message [ 2553.882178] Lustre: Skipped 1 previous similar message [ 2558.949181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2558.958346] Lustre: Skipped 2 previous similar messages [ 2559.533393] Lustre: server umount lustre-MDT0000 complete [ 2564.071627] LustreError: 75028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.105553] LustreError: 75028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2564.408191] LustreError: 78496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970945 with bad export cookie 265192139465325918 [ 2564.415899] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2564.429800] LustreError: 78496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2564.893335] Lustre: server umount lustre-MDT0001 complete [ 2577.190577] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2580.490293] Key type lgssc unregistered [ 2580.854683] LNet: 79900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2580.866507] LNetError: 79900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2580.882730] LNet: Removed LNI 192.168.204.116@tcp [ 2582.070258] Key type .llcrypt unregistered [ 2582.072760] Key type ._llcrypt unregistered [ 2596.623725] Key type ._llcrypt registered [ 2596.626465] Key type .llcrypt registered [ 2596.761576] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2597.186798] Key type .llcrypt unregistered [ 2597.196670] Key type ._llcrypt unregistered [ 2625.437197] Key type ._llcrypt registered [ 2625.440236] Key type .llcrypt registered [ 2625.581791] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2626.514546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2626.533096] alg: No test for adler32 (adler32-zlib) [ 2627.589863] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 2627.760733] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2629.447195] Key type lgssc registered [ 2630.510904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2639.201862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2639.233261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.714465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2645.796722] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2650.083994] LustreError: 81805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.106732] LustreError: 81805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2654.068582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.453577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2658.672055] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2667.506386] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2674.035244] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2683.485893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2683.960233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2686.015786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2691.470743] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2700.875577] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2714.601961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.602775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2714.629548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2714.637798] Lustre: Skipped 1 previous similar message [ 2718.183391] Lustre: server umount lustre-OST0000 complete [ 2724.834499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2724.840901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2724.849015] Lustre: Skipped 2 previous similar messages [ 2724.859110] Lustre: Skipped 1 previous similar message [ 2729.953349] LustreError: 81806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2729.962571] LustreError: 81806:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2730.706830] Lustre: server umount lustre-MDT0000 complete [ 2735.072591] LustreError: 82852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.096924] LustreError: 82852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2735.249954] LustreError: 81785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783971116 with bad export cookie 3536802557449676833 [ 2735.264370] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2735.274178] LustreError: 81785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2735.678298] Lustre: server umount lustre-MDT0001 complete [ 2748.778695] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2751.689580] Key type lgssc unregistered [ 2751.997663] LNet: 85342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2752.003853] LNetError: 85342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2752.037495] LNet: Removed LNI 192.168.204.116@tcp [ 2752.948450] Key type .llcrypt unregistered [ 2752.949781] Key type ._llcrypt unregistered [ 2781.204796] Key type ._llcrypt registered [ 2781.207626] Key type .llcrypt registered [ 2781.337916] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2782.556910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2782.642450] alg: No test for adler32 (adler32-zlib) [ 2783.687415] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 2783.933367] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2785.615238] Key type lgssc registered [ 2786.632266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2794.410312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2794.442353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.002766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2800.251404] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2805.220239] LustreError: 86755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.304339] LustreError: 86755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2810.296729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2810.338092] LustreError: 86756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.944908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2817.401727] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2827.595262] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2835.148712] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2843.538053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2843.930963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2851.322146] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2860.311992] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2866.488292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2871.776196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2871.792503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2871.799281] LustreError: Skipped 1 previous similar message [ 2871.800603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2871.831738] Lustre: Skipped 1 previous similar message [ 2871.847928] Lustre: Skipped 1 previous similar message [ 2873.637770] Lustre: server umount lustre-OST0000 complete [ 2882.020816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.044290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2882.054282] Lustre: Skipped 1 previous similar message [ 2883.888431] Lustre: server umount lustre-MDT0000 complete [ 2887.143580] LustreError: 86756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.171700] LustreError: 86756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2888.746831] LustreError: 86735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783971269 with bad export cookie 4521572225810971544 [ 2888.760613] LustreError: 86735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2888.761734] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2889.116840] Lustre: server umount lustre-MDT0001 complete [ 2900.306642] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2902.978054] Key type lgssc unregistered [ 2903.206476] LNet: 90289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2903.213771] LNetError: 90289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2903.233083] LNet: Removed LNI 192.168.204.116@tcp [ 2904.012253] Key type .llcrypt unregistered [ 2904.020805] Key type ._llcrypt unregistered [ 2933.255740] Key type ._llcrypt registered [ 2933.258681] Key type .llcrypt registered [ 2933.422737] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2934.115421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2934.136981] alg: No test for adler32 (adler32-zlib) [ 2935.285869] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 2935.524689] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2937.200073] Key type lgssc registered [ 2938.587360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2947.062078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2947.081930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.549589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.615422] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2958.835092] LustreError: 91704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.879038] LustreError: 91704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2963.552643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.937325] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2964.295368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2970.012315] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2977.983667] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2983.537432] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2990.460244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2990.730350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.838457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2996.920646] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3004.486967] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3019.751586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3019.755589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3019.783228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3019.795407] Lustre: Skipped 1 previous similar message [ 3021.058949] Lustre: server umount lustre-OST0000 complete [ 3025.169104] Lustre: server umount lustre-MDT0000 complete [ 3025.891220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3025.901978] LustreError: Skipped 1 previous similar message [ 3025.910373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.950980] LustreError: 91703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3029.354276] LustreError: 91685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783971410 with bad export cookie 14339661137815221340 [ 3029.365138] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3029.368133] LustreError: 91685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3031.807990] Lustre: server umount lustre-MDT0001 complete [ 3043.691801] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 3046.617665] Key type lgssc unregistered [ 3046.946638] LNet: 95259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3046.961392] LNetError: 95259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3046.977408] LNet: Removed LNI 192.168.204.116@tcp [ 3048.098684] Key type .llcrypt unregistered [ 3048.101196] Key type ._llcrypt unregistered [ 3073.916798] Key type ._llcrypt registered [ 3073.919206] Key type .llcrypt registered [ 3074.030318] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3075.494650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3075.545204] alg: No test for adler32 (adler32-zlib) [ 3076.612826] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 3076.862674] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 3078.535605] Key type lgssc registered [ 3079.535639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3086.447709] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3086.482050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3102.112953] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3107.231903] LustreError: 96651:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 3133.921500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.142662] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3147.969101] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3154.125297] Lustre: server umount lustre-OST0000 complete [ 3163.625173] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 15:39:02 (1783971542) [ 3171.072981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.657022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.927392] LustreError: 98290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3176.934479] LustreError: 98290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3177.202858] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3182.058460] LustreError: 98291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3187.169705] LustreError: 98290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3187.907852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3188.286353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3193.674448] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3202.609900] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3208.869927] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3216.234237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3216.609851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.646020] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3223.076570] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3230.222907] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3244.518953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.526764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3244.531275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3249.635599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3249.642840] Lustre: Skipped 2 previous similar messages [ 3249.856487] Lustre: server umount lustre-OST0000 complete [ 3254.755265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.764297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3254.768745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3254.784769] Lustre: Skipped 2 previous similar messages [ 3254.815612] Lustre: Skipped 1 previous similar message [ 3259.688108] Lustre: server umount lustre-MDT0000 complete [ 3259.880751] LustreError: 98291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3259.896143] LustreError: 98291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3262.926983] LustreError: 98271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783971643 with bad export cookie 11957967824197451234 [ 3262.929026] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3262.944681] LustreError: 98271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3263.397246] Lustre: server umount lustre-MDT0001 complete [ 3272.714980] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 3275.375836] Key type lgssc unregistered [ 3275.613876] LNet: 101980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3275.623184] LNetError: 101980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3275.650609] LNet: Removed LNI 192.168.204.116@tcp [ 3276.423907] Key type .llcrypt unregistered [ 3276.429118] Key type ._llcrypt unregistered [ 3316.865871] Key type ._llcrypt registered [ 3316.869546] Key type .llcrypt registered [ 3317.019186] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3318.178937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3318.353417] alg: No test for adler32 (adler32-zlib) [ 3319.447626] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 3319.728060] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 3321.431201] Key type lgssc registered [ 3322.752039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3330.948915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3330.962932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.497296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3337.130439] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3342.821422] LustreError: 104168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.837978] LustreError: 104168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3345.683435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.176143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3351.032286] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3358.295349] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3363.734536] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3369.956277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3370.257247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3377.155418] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3377.472269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3387.198358] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3398.113566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.116392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3398.154828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3403.235493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3403.244020] Lustre: Skipped 1 previous similar message [ 3404.158296] Lustre: server umount lustre-OST0000 complete [ 3408.354021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3408.361203] Lustre: Skipped 1 previous similar message [ 3408.364074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3408.368319] Lustre: Skipped 1 previous similar message [ 3413.473582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.480201] Lustre: Skipped 1 previous similar message [ 3414.454545] Lustre: server umount lustre-MDT0000 complete [ 3417.497588] LustreError: 104148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783971798 with bad export cookie 15466276154059830517 [ 3417.501537] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3417.519635] LustreError: 104148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3417.816552] Lustre: server umount lustre-MDT0001 complete [ 3427.546377] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 3430.241362] Key type lgssc unregistered [ 3430.504212] LNet: 107703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3430.514601] LNetError: 107703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3430.530844] LNet: Removed LNI 192.168.204.116@tcp [ 3431.357717] Key type .llcrypt unregistered [ 3431.363233] Key type ._llcrypt unregistered [ 3449.694520] Key type ._llcrypt registered [ 3449.702647] Key type .llcrypt registered [ 3449.821178] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3450.870890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3450.971746] alg: No test for adler32 (adler32-zlib) [ 3452.032822] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 3452.296230] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 3453.999216] Key type lgssc registered [ 3455.216712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3462.683476] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3462.700320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3478.303604] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3483.423972] LustreError: 108956:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 3510.175288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3516.790437] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3524.922937] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3531.906558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3532.199053] LustreError: MGC192.168.204.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3532.207663] LustreError: 109980:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3532.216251] LustreError: 109980:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3532.223398] LustreError: 109980:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3532.329113] Lustre: server umount lustre-MDT0000 complete [ 3532.333518] LustreError: 109980:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3553.761047] Lustre: 108294:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783971918/real 1783971918] req@00000000490ddba8 x1870630048763136/t0(0) o400->MGC192.168.204.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783971934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3553.791448] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3553.808396] LustreError: 108968:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3554.159976] Lustre: server umount lustre-OST0000 complete [ 3564.053966] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 3574.983761] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3611.144927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3611.447044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3611.492389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3611.610961] Lustre: lustre-MDT0000: new disk, initializing [ 3611.708750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3611.742209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3616.156960] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3627.228988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.388214] Lustre: 112772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3627.446639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3627.454531] Lustre: Skipped 1 previous similar message [ 3627.521338] Lustre: lustre-MDT0001: new disk, initializing [ 3627.618132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3627.650317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3627.667131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3631.823333] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3640.644503] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3646.327489] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3653.224717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3653.466826] Lustre: lustre-OST0000: new disk, initializing [ 3653.478412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3653.487473] Lustre: 114739:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3653.594443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.382036] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3662.898881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3662.907175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3663.009856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3669.338793] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3674.657279] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3674.863648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3680.246662] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3680.485428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3683.820277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3683.821531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.828769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3683.860137] Lustre: Skipped 1 previous similar message [ 3688.575929] Lustre: server umount lustre-OST0000 complete [ 3694.048930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3694.048958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.068018] LustreError: Skipped 1 previous similar message [ 3694.088052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3694.088060] Lustre: Skipped 1 previous similar message [ 3698.994966] Lustre: server umount lustre-MDT0000 complete [ 3702.724752] LustreError: 112764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972083 with bad export cookie 1092246971865869909 [ 3702.732083] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3703.036687] Lustre: server umount lustre-MDT0001 complete [ 3711.561235] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 15:48:10 (1783972090) [ 3715.179391] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3730.991224] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 3741.580596] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3778.799618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3778.997719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3779.030526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3779.100694] Lustre: lustre-MDT0000: new disk, initializing [ 3779.196613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3779.228368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3784.047836] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3795.249155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3795.381061] Lustre: 119643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3795.445850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3795.449842] Lustre: Skipped 1 previous similar message [ 3795.539766] Lustre: lustre-MDT0001: new disk, initializing [ 3795.635768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3795.665339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3795.678749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3800.131548] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3811.260847] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3817.739348] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3824.799498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3825.129082] Lustre: lustre-OST0000: new disk, initializing [ 3825.138091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3825.148264] Lustre: 121611:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3825.216476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3830.845357] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3831.881955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3831.893486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3832.013618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3839.877664] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3843.966397] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3844.161311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3848.590995] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3848.815186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3852.768415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3852.772831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3852.782144] LustreError: Skipped 1 previous similar message [ 3852.782843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3852.782847] Lustre: Skipped 1 previous similar message [ 3852.812199] Lustre: Skipped 2 previous similar messages [ 3857.897558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3857.901443] Lustre: Skipped 1 previous similar message [ 3863.018590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3863.023398] Lustre: Skipped 1 previous similar message [ 3865.055150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3865.243752] Lustre: server umount lustre-OST0000 complete [ 3872.736570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3872.752145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3872.768817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3872.771997] Lustre: Skipped 1 previous similar message [ 3875.337306] Lustre: server umount lustre-MDT0000 complete [ 3878.371505] LustreError: 119651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3878.390595] LustreError: 119651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3878.530392] LustreError: 119635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972259 with bad export cookie 1092246971865870791 [ 3878.534047] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3878.539560] LustreError: 119635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3878.801176] Lustre: server umount lustre-MDT0001 complete [ 3886.694661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3887.049554] LustreError: 123825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3887.125946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3891.170918] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3892.199532] LustreError: 123826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3897.316289] LustreError: 123825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.199648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3905.267651] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3912.842621] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3919.187591] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3926.570301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3933.653364] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3941.452361] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3952.669346] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3958.756437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3958.770175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3958.793367] Lustre: Skipped 1 previous similar message [ 3958.803512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3958.808523] Lustre: Skipped 2 previous similar messages [ 3963.046526] Lustre: server umount lustre-OST0000 complete [ 3970.016312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3970.033698] Lustre: Skipped 2 previous similar messages [ 3973.442959] Lustre: server umount lustre-MDT0000 complete [ 3975.136481] LustreError: 123826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.159864] LustreError: 123826:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3976.794404] LustreError: 123807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972357 with bad export cookie 1092246971865871673 [ 3976.800369] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3976.801360] LustreError: 123807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3977.172279] Lustre: server umount lustre-MDT0001 complete [ 3988.315652] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 3991.678170] Key type lgssc unregistered [ 3991.943849] LNet: 127457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3991.958931] LNetError: 127457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3991.996615] LNet: Removed LNI 192.168.204.116@tcp [ 3992.771260] Key type .llcrypt unregistered [ 3992.774073] Key type ._llcrypt unregistered [ 4009.661236] Key type ._llcrypt registered [ 4009.662962] Key type .llcrypt registered [ 4009.811263] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 4024.524261] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 4025.375631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4025.414068] alg: No test for adler32 (adler32-zlib) [ 4026.561156] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 4026.819921] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 4028.482657] Key type lgssc registered [ 4029.641244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4067.343564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4067.392211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4068.659877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4068.687121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4068.783460] Lustre: lustre-MDT0000: new disk, initializing [ 4068.884633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.915666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4073.307654] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4083.605385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4083.720748] Lustre: 130611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4083.792903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4083.795893] Lustre: Skipped 1 previous similar message [ 4083.840746] Lustre: lustre-MDT0001: new disk, initializing [ 4083.923104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4083.977690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4083.987028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4090.104519] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4100.688456] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4106.240139] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4112.961167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4113.216292] Lustre: lustre-OST0000: new disk, initializing [ 4113.219748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4113.227392] Lustre: 132580:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4113.315785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4120.149353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4120.175759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4120.324810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4120.428169] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4130.857683] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4136.085590] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4136.356577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4141.588831] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4141.818381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4146.150792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4146.151762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.161907] LustreError: Skipped 1 previous similar message [ 4146.163579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4146.206148] Lustre: Skipped 1 previous similar message [ 4150.263506] Lustre: server umount lustre-OST0000 complete [ 4155.872258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4155.879419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4155.896651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4155.901129] Lustre: Skipped 1 previous similar message [ 4160.596507] Lustre: server umount lustre-MDT0000 complete [ 4164.798579] LustreError: 130605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972545 with bad export cookie 16656435910513140907 [ 4164.801350] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4164.808672] LustreError: 130605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4165.320635] Lustre: server umount lustre-MDT0001 complete [ 4174.650000] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 15:55:53 (1783972553) [ 4181.449125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4181.911930] LustreError: 134994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.928675] LustreError: 134994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4182.000570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4186.692449] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4187.110122] LustreError: 134993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4191.203862] LustreError: 134994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.739993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4196.039050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4200.533816] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4208.812402] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4215.420119] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4220.451534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4220.457559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4220.466271] Lustre: Skipped 2 previous similar messages [ 4220.473376] Lustre: Skipped 3 previous similar messages [ 4223.942298] Lustre: server umount lustre-MDT0000 complete [ 4225.505586] LustreError: 134994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4225.531829] LustreError: 134994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4227.962731] LustreError: 134973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972608 with bad export cookie 16656435910513141789 [ 4227.999847] LustreError: 134973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4228.000586] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4228.314440] Lustre: server umount lustre-MDT0001 complete [ 4237.231159] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 15:56:56 (1783972616) [ 4243.365136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4258.975558] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4264.095950] LustreError: 137408:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 4289.823913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4296.670976] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4303.829568] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4306.548990] Lustre: server umount lustre-OST0000 complete [ 4316.309685] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 15:58:15 (1783972695) [ 4323.189466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4323.668746] LustreError: 138854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4323.751448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4328.562903] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4334.059715] LustreError: 138854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4334.082529] LustreError: 138854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4338.658997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4343.509936] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4352.205335] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4358.833217] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4365.543577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4365.908954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.916975] Lustre: Skipped 1 previous similar message [ 4372.026685] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4379.839634] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4393.961038] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4393.963145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4393.995357] Lustre: Skipped 2 previous similar messages [ 4394.007393] Lustre: Skipped 2 previous similar messages [ 4398.821029] Lustre: server umount lustre-MDT0000 complete [ 4399.075615] LustreError: 138854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4399.109378] LustreError: 138854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4402.705536] LustreError: 140701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972783 with bad export cookie 16656435910513142454 [ 4402.709240] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4402.713126] LustreError: 140701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4402.960534] Lustre: server umount lustre-MDT0001 complete [ 4417.181569] Lustre: server umount lustre-OST0000 complete [ 4425.199039] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 16:00:04 (1783972804) [ 4431.538249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.091825] LustreError: 142534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4432.180931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4436.671649] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4445.674684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4449.905882] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4457.190071] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4463.167989] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4470.768563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4477.435182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4478.136858] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4486.345041] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4490.662366] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4490.861089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4494.319753] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4494.515436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4497.892351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4497.902150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4497.910202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4500.964909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4502.649573] Lustre: server umount lustre-OST0000 complete [ 4507.616646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4507.629531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.646565] Lustre: Skipped 1 previous similar message [ 4507.655454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4507.658015] Lustre: Skipped 2 previous similar messages [ 4513.123982] Lustre: server umount lustre-MDT0000 complete [ 4516.332644] LustreError: 142531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.348431] LustreError: 142531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4517.644046] LustreError: 142515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972898 with bad export cookie 16656435910513143525 [ 4517.651260] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4517.664689] LustreError: 142515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4518.376995] Lustre: server umount lustre-MDT0001 complete [ 4529.057200] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 16:01:47 (1783972907) [ 4535.960700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4551.519604] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4556.639838] LustreError: 146706:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 4582.367908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4582.385326] Lustre: Skipped 2 previous similar messages [ 4588.549715] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4595.934504] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4602.810132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4602.953298] LustreError: 128833:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000040fb37af x1870630650770304/t0(0) o250->MGC192.168.204.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4603.366309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4608.353231] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4618.186876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.001875] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4634.306614] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4640.529907] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4646.078164] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4646.296423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4651.099230] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4651.352786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4654.563848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4654.566905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4654.572957] Lustre: Skipped 3 previous similar messages [ 4654.573436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4654.594096] Lustre: Skipped 4 previous similar messages [ 4659.579349] Lustre: server umount lustre-MDT0000 complete [ 4659.687378] LustreError: 147743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4659.714428] LustreError: 147743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4663.531255] LustreError: 146714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973044 with bad export cookie 16656435910513144547 [ 4663.540785] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4663.554223] LustreError: 146714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4664.004637] Lustre: server umount lustre-MDT0001 complete [ 4678.317695] Lustre: server umount lustre-OST0000 complete [ 4687.568312] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 16:04:26 (1783973066) [ 4695.304688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4710.879661] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4715.999788] LustreError: 150898:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 4749.132735] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4757.590516] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4764.782878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.962636] LustreError: 128833:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a8f3ccc2 x1870630650799744/t0(0) o250->MGC192.168.204.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4765.432639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4769.773336] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4779.061526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.424550] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4792.109825] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4798.061212] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4804.523035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4804.713791] Lustre: lustre-OST0001: new disk, initializing [ 4804.719327] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4804.726239] Lustre: 153769:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4810.909670] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4811.868990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4811.881104] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4811.985311] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4821.231257] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4826.143595] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4826.377928] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4831.687722] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4831.897423] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4835.812163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4835.830106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4835.864892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4840.159883] Lustre: server umount lustre-OST0000 complete [ 4858.335862] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4858.608818] Lustre: server umount lustre-OST0001 complete [ 4877.279147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4877.522869] Lustre: server umount lustre-MDT0000 complete [ 4881.486343] LustreError: 150906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973262 with bad export cookie 16656435910513145499 [ 4881.499913] LustreError: 150906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4881.500161] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4881.945508] Lustre: server umount lustre-MDT0001 complete [ 4906.409239] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 16:08:05 (1783973285) [ 4908.144516] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4909.925959] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4911.768554] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 16:08:10 (1783973290) [ 4918.662088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4919.043178] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4919.063321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4919.192048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4919.195676] Lustre: Skipped 6 previous similar messages [ 4923.475246] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4931.128475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4931.240388] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4931.279481] Lustre: 157333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4935.997359] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4943.153273] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4949.039478] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4955.235233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4955.440342] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4961.218402] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4965.543546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4968.768573] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4973.347917] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4973.523868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4977.906484] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4978.153710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4980.705567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4980.710828] LustreError: Skipped 3 previous similar messages [ 4980.714965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4980.725977] Lustre: Skipped 5 previous similar messages [ 4980.729519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4980.732256] Lustre: Skipped 14 previous similar messages [ 4994.015351] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4994.246384] Lustre: server umount lustre-OST0000 complete [ 5017.972266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5021.503466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.521724] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.525628] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5025.637880] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5034.050877] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5039.344461] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5039.587981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5044.671476] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5045.009193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5051.411648] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff91b207600800.ost_server_uuid 50 [ 5052.963548] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91b207600800.ost_server_uuid in FULL state after 0 sec [ 5053.140732] LustreError: 157341:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.16@tcp) returned error from blocking AST (req@00000000214f4575 x1870630650910592 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000005d3585dc/0xe7278c3606323ff5 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.16@tcp remote: 0x37979ca7b1621a0 expref: 5 pid: 157339 timeout: 5153 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5053.179686] LustreError: lustre-MDT0000: A client on nid 192.168.204.16@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5053.187678] LustreError: 157329:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.16@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000005d3585dc/0xe7278c3606323ff5 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.16@tcp remote: 0x37979ca7b1621a0 expref: 6 pid: 157339 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5057.516885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5057.527118] LustreError: Skipped 1 previous similar message [ 5063.677737] Lustre: server umount lustre-OST0000 complete [ 5074.331396] Lustre: server umount lustre-MDT0000 complete [ 5077.986525] LustreError: 158299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5078.025540] LustreError: 158299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 5078.389677] LustreError: 158084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973459 with bad export cookie 16656435910513146682 [ 5078.393085] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5078.402932] LustreError: 158084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5085.031993] Lustre: server umount lustre-MDT0001 complete [ 5098.507796] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5101.167298] Key type lgssc unregistered [ 5101.465542] LNet: 163450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5101.470784] LNetError: 163450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5101.480852] LNet: Removed LNI 192.168.204.116@tcp [ 5102.083856] Key type .llcrypt unregistered [ 5102.086599] Key type ._llcrypt unregistered [ 5132.601510] Key type ._llcrypt registered [ 5132.602911] Key type .llcrypt registered [ 5132.802835] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 5134.574390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5134.608725] alg: No test for adler32 (adler32-zlib) [ 5135.850253] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 5136.155835] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 5137.864193] Key type lgssc registered [ 5139.356988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5150.160418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5150.185653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5151.645611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.086652] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5161.957438] LustreError: 164866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5161.968784] LustreError: 164866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5163.865943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.203250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5167.974365] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5175.692330] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5181.700260] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5188.173890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5188.530728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5194.259493] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5196.744616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5202.623784] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5210.126169] Lustre: Failing over lustre-MDT0000 [ 5210.465646] Lustre: server umount lustre-MDT0000 complete [ 5210.928955] LustreError: 164863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5212.133811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5212.149112] LustreError: 165939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5212.159489] Lustre: Skipped 2 previous similar messages [ 5212.200479] LustreError: 165939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5217.250829] LustreError: 164861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5217.282395] LustreError: 164861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5222.370032] LustreError: 165939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5222.390747] LustreError: 165939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5223.202189] LustreError: 167445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.16@tcp arrived at 1783973604 with bad export cookie 7125641767520384044 [ 5223.217831] LustreError: 167445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5228.450378] Lustre: 164175:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783973593/real 1783973593] req@000000002b13b147 x1870631814588544/t0(0) o400->MGC192.168.204.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783973609 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5228.483988] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5231.162313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5233.636099] LustreError: 164866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5233.657575] LustreError: 164866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5239.775872] LustreError: 164174:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000098c2bc04 x1870631814594688/t0(0) o250->MGC192.168.204.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5240.057533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5240.125549] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5244.603553] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5245.425059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5245.426561] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5247.006483] Lustre: lustre-MDT0000: Denying connection for new client 225731f8-06a0-43c5-8f0d-2208cb86732f (at 192.168.204.16@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5255.648993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.668888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5255.674289] Lustre: Skipped 1 previous similar message [ 5260.772320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5260.780059] Lustre: Skipped 1 previous similar message [ 5261.657193] Lustre: server umount lustre-OST0000 complete [ 5265.112739] LustreError: 168598:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5265.122980] Lustre: 167945:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5265.133928] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5265.155540] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5265.167627] LustreError: 167945:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000046094fc x1870631814609280/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5265.186470] LustreError: 167945:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5265.193452] LustreError: 167945:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5265.199616] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5265.763473] Lustre: server umount lustre-MDT0000 complete [ 5265.904139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.935216] Lustre: Skipped 1 previous similar message [ 5265.950071] LustreError: 164863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5265.984759] LustreError: 164863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5269.710463] LustreError: 167445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973650 with bad export cookie 7125641767520384142 [ 5269.717515] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5269.726887] LustreError: 167445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5269.746467] LustreError: 164174:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5272.105696] Lustre: server umount lustre-MDT0001 complete [ 5283.770976] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5286.561315] Key type lgssc unregistered [ 5286.981934] LNet: 169350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5287.003360] LNetError: 169350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5287.037177] LNet: Removed LNI 192.168.204.116@tcp [ 5288.209204] Key type .llcrypt unregistered [ 5288.214575] Key type ._llcrypt unregistered [ 5312.796416] Key type ._llcrypt registered [ 5312.802226] Key type .llcrypt registered [ 5312.914514] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 5314.323486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5314.347493] alg: No test for adler32 (adler32-zlib) [ 5315.517609] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 5315.756919] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 5317.399276] Key type lgssc registered [ 5318.556816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5326.791591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5326.809776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.198621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5332.920338] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5338.594635] LustreError: 170763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5338.605391] LustreError: 170763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5341.320052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5341.623563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5345.261689] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5352.199259] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5358.223659] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5365.674078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5366.061565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5371.580333] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5373.240121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5379.031313] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5387.398400] Lustre: server umount lustre-OST0000 complete [ 5392.868733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5392.882637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5392.904802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5393.895808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5393.898090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.013921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5399.027508] Lustre: Skipped 2 previous similar messages [ 5402.179112] Lustre: server umount lustre-MDT0000 complete [ 5403.103428] Lustre: 170073:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783973768/real 1783973768] req@000000002e693925 x1870632002283776/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783973784 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5403.138867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5403.154109] LustreError: 172666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5403.180412] LustreError: 172666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5404.191280] Lustre: 170073:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783973769/real 1783973769] req@00000000a0efc18f x1870632002284672/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783973785 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5406.396065] LustreError: 173543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973787 with bad export cookie 7283970225064888736 [ 5406.405904] LustreError: 173543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5406.406893] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5406.838354] Lustre: server umount lustre-MDT0001 complete [ 5418.862510] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5422.258810] Key type lgssc unregistered [ 5422.702606] LNet: 174297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5422.711971] LNetError: 174297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5422.749993] LNet: Removed LNI 192.168.204.116@tcp [ 5423.610578] Key type .llcrypt unregistered [ 5423.615131] Key type ._llcrypt unregistered [ 5459.434211] Key type ._llcrypt registered [ 5459.437382] Key type .llcrypt registered [ 5459.519230] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 5461.034408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5461.062676] alg: No test for adler32 (adler32-zlib) [ 5462.187272] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 5462.442376] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 5464.135277] Key type lgssc registered [ 5465.255310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5473.100670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5473.127430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5474.549322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5478.276500] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5485.026099] LustreError: 175999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5485.045738] LustreError: 175999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5485.335298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5485.707531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5490.046979] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5499.462251] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5506.322846] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5513.799842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5514.160352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.297490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5520.508357] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5528.379111] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5535.713484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5535.713743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5535.717063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5535.722490] Lustre: Skipped 1 previous similar message [ 5537.324467] Lustre: server umount lustre-OST0000 complete [ 5542.370340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5542.381216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.392228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5542.397161] Lustre: Skipped 1 previous similar message [ 5545.953928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5545.960345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5547.502731] Lustre: server umount lustre-MDT0000 complete [ 5551.081610] LustreError: 177024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5551.105248] LustreError: 177024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5551.627916] LustreError: 175979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973932 with bad export cookie 4163141119130012050 [ 5551.632713] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5551.641565] LustreError: 175979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5552.011122] Lustre: server umount lustre-MDT0001 complete [ 5561.756655] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5564.120579] Key type lgssc unregistered [ 5564.449882] LNet: 179528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5564.458157] LNetError: 179528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5564.476378] LNet: Removed LNI 192.168.204.116@tcp [ 5565.439719] Key type .llcrypt unregistered [ 5565.448192] Key type ._llcrypt unregistered [ 5585.219669] Key type ._llcrypt registered [ 5585.221486] Key type .llcrypt registered [ 5585.320196] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 5586.398878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5586.444227] alg: No test for adler32 (adler32-zlib) [ 5587.507620] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 5587.766618] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 5589.591232] Key type lgssc registered [ 5590.348664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5601.614993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5601.639103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.078126] Lustre: *** cfs_fail_loc=135, val=0*** [ 5603.243888] LustreError: 181032:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5603.261407] LustreError: 181032:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.116@tcp: cfg command failed: rc = -2 [ 5603.274044] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5603.274044] [ 5603.282885] LustreError: MGC192.168.204.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5603.304141] LustreError: 180993:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5603.318319] LustreError: 180993:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5603.326654] LustreError: 180993:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5603.333901] LustreError: 180993:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5603.444931] Lustre: server umount lustre-MDT0000 complete [ 5603.449039] LustreError: 180993:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5609.709072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5610.184107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.221905] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5620.708825] LustreError: 181403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5620.733395] LustreError: 181403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5624.551936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5624.965995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5629.501463] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5638.466847] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5647.089903] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5650.400718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5650.415803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5650.441027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5655.019730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5655.030146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.631872] Lustre: server umount lustre-MDT0000 complete [ 5660.129251] LustreError: 181404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5660.150794] LustreError: 181404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5660.430726] LustreError: 181386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974041 with bad export cookie 11169618480724597997 [ 5660.438414] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5660.453957] LustreError: 181386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5660.726924] Lustre: server umount lustre-MDT0001 complete [ 5671.760769] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5675.060857] Key type lgssc unregistered [ 5675.380878] LNet: 183725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5675.397502] LNetError: 183725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5675.421157] LNet: Removed LNI 192.168.204.116@tcp [ 5676.561364] Key type .llcrypt unregistered [ 5676.565779] Key type ._llcrypt unregistered [ 5700.268514] Key type ._llcrypt registered [ 5700.269905] Key type .llcrypt registered [ 5700.362597] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5700.679317] Key type .llcrypt unregistered [ 5700.686787] Key type ._llcrypt unregistered [ 5715.975701] Key type ._llcrypt registered [ 5715.980879] Key type .llcrypt registered [ 5716.096655] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 5717.142133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5717.209758] alg: No test for adler32 (adler32-zlib) [ 5718.323260] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 5718.731601] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 5720.455455] Key type lgssc registered [ 5721.513743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5728.882631] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5728.908168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5744.543702] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5749.664119] LustreError: 185870:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 5776.351392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5783.062649] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5790.106267] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5797.314446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.918724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5798.014853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5802.396784] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5802.986341] LustreError: 185895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5803.011253] LustreError: 185895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5808.098015] LustreError: 185896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5810.159950] LustreError: 185895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5810.182127] LustreError: 185895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5812.325186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5812.834275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5817.717845] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5826.362436] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5832.833803] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5836.148707] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5848.567358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5848.582877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.592206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5848.621127] Lustre: Skipped 1 previous similar message [ 5852.858384] Lustre: server umount lustre-OST0000 complete [ 5858.784388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.790662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5858.817772] Lustre: Skipped 1 previous similar message [ 5858.818470] Lustre: Skipped 1 previous similar message [ 5863.314354] Lustre: server umount lustre-MDT0000 complete [ 5863.918077] LustreError: 186910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5863.943283] LustreError: 186910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5867.115814] LustreError: 188882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974248 with bad export cookie 6977834347299240620 [ 5867.118446] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5867.123352] LustreError: 188882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5867.415579] Lustre: server umount lustre-MDT0001 complete [ 5879.484216] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5882.226844] Key type lgssc unregistered [ 5882.480730] LNet: 189835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5882.492092] LNetError: 189835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5882.524396] LNet: Removed LNI 192.168.204.116@tcp [ 5883.396211] Key type .llcrypt unregistered [ 5883.401909] Key type ._llcrypt unregistered [ 5911.364816] Key type ._llcrypt registered [ 5911.368127] Key type .llcrypt registered [ 5911.474709] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 5912.723486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5912.768251] alg: No test for adler32 (adler32-zlib) [ 5913.785247] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 5914.056073] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 5915.751242] Key type lgssc registered [ 5916.874839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5925.330257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5925.357908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5927.148635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5933.067721] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5936.616375] LustreError: 191246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5936.644221] LustreError: 191246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5941.730573] LustreError: 191247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5942.675051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5943.238485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5948.169584] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5955.727143] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5963.858087] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5971.899637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5972.289204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5979.239417] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5988.087858] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5996.857041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6004.626340] Lustre: Failing over lustre-MDT0000 [ 6004.706349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6004.712889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.731806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6005.520490] LustreError: 191243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6006.864661] Lustre: server umount lustre-MDT0000 complete [ 6007.273177] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6007.290893] Lustre: Skipped 1 previous similar message [ 6010.610118] LustreError: 192249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6010.664117] LustreError: 192249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6015.736102] LustreError: 191242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6015.758567] LustreError: 191242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6023.655987] Lustre: 190557:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783974388/real 1783974388] req@0000000025de604b x1870632629335808/t0(0) o400->MGC192.168.204.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783974404 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6023.685682] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6025.969795] LustreError: 191240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6025.991079] LustreError: 191240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6026.199358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6034.198864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6034.253784] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6036.208454] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6038.855579] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6039.538187] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6039.579654] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6039.636131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6043.320550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6052.984839] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6067.174911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6067.188317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6067.211374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6070.243424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6070.249281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6072.927711] Lustre: server umount lustre-OST0000 complete [ 6080.481723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6080.482544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6080.492854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6080.505829] Lustre: Skipped 1 previous similar message [ 6080.509194] Lustre: Skipped 1 previous similar message [ 6083.153943] Lustre: server umount lustre-MDT0000 complete [ 6085.613874] LustreError: 193741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6085.637113] LustreError: 193741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6087.344151] LustreError: 191981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974468 with bad export cookie 6452631868576500199 [ 6087.352000] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6087.358025] LustreError: 191981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6087.856171] Lustre: server umount lustre-MDT0001 complete [ 6099.966801] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 6103.574725] Key type lgssc unregistered [ 6103.943317] LNet: 196228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6103.949126] LNetError: 196228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6103.996038] LNet: Removed LNI 192.168.204.116@tcp [ 6105.309120] Key type .llcrypt unregistered [ 6105.314245] Key type ._llcrypt unregistered [ 6133.739592] Key type ._llcrypt registered [ 6133.742101] Key type .llcrypt registered [ 6133.857404] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 6134.772825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6134.788877] alg: No test for adler32 (adler32-zlib) [ 6135.931755] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 6136.150027] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 6137.913150] Key type lgssc registered [ 6139.194327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6148.283175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6148.329796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6149.873642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6154.600852] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6159.330861] LustreError: 197641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6159.347584] LustreError: 197641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6164.225852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6164.455763] LustreError: 197640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6164.862687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6170.058960] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6177.887662] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6183.843504] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6192.716105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6193.229207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6194.344453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6201.570217] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6210.679371] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6214.762693] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6221.968399] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6241.788117] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6256.099638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6256.102634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6256.102611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6256.103332] LustreError: Skipped 1 previous similar message [ 6256.111834] Lustre: Skipped 1 previous similar message [ 6258.714389] Lustre: server umount lustre-OST0000 complete [ 6262.241085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6262.248393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6262.276293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6262.286092] Lustre: Skipped 1 previous similar message [ 6266.338110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6266.340511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6268.302424] Lustre: server umount lustre-MDT0000 complete [ 6271.458591] LustreError: 199542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6271.496535] LustreError: 199542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6272.471215] LustreError: 197620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974653 with bad export cookie 10204221744622641691 [ 6272.476524] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6272.483394] LustreError: 197620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6273.064405] Lustre: server umount lustre-MDT0001 complete [ 6287.429390] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 6290.469311] Key type lgssc unregistered [ 6290.882892] LNet: 201508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6290.894350] LNetError: 201508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6291.943633] LNet: Removed LNI 192.168.204.116@tcp [ 6292.828325] Key type .llcrypt unregistered [ 6292.830159] Key type ._llcrypt unregistered [ 6316.779909] Key type ._llcrypt registered [ 6316.781784] Key type .llcrypt registered [ 6316.879536] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 6318.744514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6318.775513] alg: No test for adler32 (adler32-zlib) [ 6319.990863] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 6320.381694] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 6322.137829] Key type lgssc registered [ 6324.256708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6333.320677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6333.346412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6334.820911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6339.056667] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6344.162200] LustreError: 202777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6344.182247] LustreError: 202777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6348.035858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6348.633405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6354.984112] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6365.073136] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6370.901138] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6378.053638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6378.324970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6379.409953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6384.709508] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6392.571812] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6398.112728] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6409.355147] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6417.411501] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6423.063082] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6434.790137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6434.793807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6434.809606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6437.859944] Lustre: server umount lustre-OST0000 complete [ 6440.928581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6440.929279] LustreError: 202088:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e627ac49 x1870633056135552/t0(0) o41->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6440.938743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.968907] Lustre: Skipped 1 previous similar message [ 6440.972769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6440.976797] Lustre: Skipped 1 previous similar message [ 6445.025210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6445.026037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6450.146531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6450.148612] Lustre: Skipped 1 previous similar message [ 6454.252396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6454.263341] Lustre: Skipped 1 previous similar message [ 6455.263787] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6455.608807] Lustre: server umount lustre-MDT0000 complete [ 6459.362336] LustreError: 202778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6459.386067] LustreError: 202778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6460.313524] LustreError: 203513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974841 with bad export cookie 10593999904559643624 [ 6460.329562] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6460.343179] LustreError: 203513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6460.666913] Lustre: server umount lustre-MDT0001 complete [ 6474.542157] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 6477.517093] Key type lgssc unregistered [ 6477.816664] LNet: 207656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6477.834470] LNetError: 207656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6477.878113] LNet: Removed LNI 192.168.204.116@tcp [ 6479.181363] Key type .llcrypt unregistered [ 6479.184517] Key type ._llcrypt unregistered [ 6502.574910] Key type ._llcrypt registered [ 6502.585712] Key type .llcrypt registered [ 6502.750706] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 6503.873304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6503.921229] alg: No test for adler32 (adler32-zlib) [ 6505.044462] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 6505.322435] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 6507.020982] Key type lgssc registered [ 6508.002037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6516.843783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6516.860928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6518.465277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6524.157345] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6529.009278] LustreError: 209068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6529.034410] LustreError: 209068:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6534.122971] LustreError: 209069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6534.594190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.000557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6539.899459] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6548.978524] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6555.416634] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6562.956126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6563.410954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6564.484438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6572.336726] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6581.542304] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6600.683097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6600.684161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6600.703873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6602.111929] Lustre: server umount lustre-OST0000 complete [ 6606.820912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6606.833804] LustreError: Skipped 1 previous similar message [ 6606.837499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.852874] Lustre: Skipped 1 previous similar message [ 6606.858919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6606.864595] Lustre: Skipped 1 previous similar message [ 6610.914369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6610.915485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6612.771417] Lustre: server umount lustre-MDT0000 complete [ 6616.036624] LustreError: 211514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6616.068200] LustreError: 211514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6617.079985] LustreError: 209049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974998 with bad export cookie 1491281498004003406 [ 6617.082858] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6617.088905] LustreError: 209049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6617.487473] Lustre: server umount lustre-MDT0001 complete [ 6631.743405] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 6635.169994] Key type lgssc unregistered [ 6635.470586] LNet: 212755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6635.500952] LNetError: 212755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6636.585948] LNet: Removed LNI 192.168.204.116@tcp [ 6637.723707] Key type .llcrypt unregistered [ 6637.726176] Key type ._llcrypt unregistered [ 6658.070705] Key type ._llcrypt registered [ 6658.073386] Key type .llcrypt registered [ 6658.186644] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 6659.288427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6659.298266] alg: No test for adler32 (adler32-zlib) [ 6660.458933] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 6660.711495] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 6662.409565] Key type lgssc registered [ 6663.824250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6673.413230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6673.436948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6675.363453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6681.144821] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6685.673779] LustreError: 214025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6685.727414] LustreError: 214025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6690.791402] LustreError: 214026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6692.407725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6692.682347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6697.883774] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6706.016905] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6712.188976] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6722.036790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6722.442443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6731.526679] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6742.373604] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6746.051240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6764.517218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6764.528791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6764.545083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6766.574075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6766.582781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6766.602932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6766.622755] Lustre: Skipped 1 previous similar message [ 6769.353675] Lustre: server umount lustre-OST0000 complete [ 6774.758500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6774.764444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6774.773983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6776.802486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6776.803213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6780.636928] Lustre: server umount lustre-MDT0000 complete [ 6781.921449] LustreError: 214025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6781.949064] LustreError: 214025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6785.056846] LustreError: 214006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783975165 with bad export cookie 2521434820794319621 [ 6785.073965] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6785.376968] Lustre: server umount lustre-MDT0001 complete [ 6796.026906] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 6799.452965] Key type lgssc unregistered [ 6799.983325] LNet: 217730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6800.010237] LNetError: 217730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6800.056633] LNet: Removed LNI 192.168.204.116@tcp [ 6801.265435] Key type .llcrypt unregistered [ 6801.268977] Key type ._llcrypt unregistered [ 6820.770526] Key type ._llcrypt registered [ 6820.776172] Key type .llcrypt registered [ 6820.962145] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 6822.293631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6822.386377] alg: No test for adler32 (adler32-zlib) [ 6823.633473] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 6823.896457] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 6825.559166] Key type lgssc registered [ 6826.976201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6835.415389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6835.431479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.876684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6841.748639] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6846.434690] LustreError: 218997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6846.466353] LustreError: 218997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6851.556096] LustreError: 218998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6852.670817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6853.199750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6858.756944] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6869.073785] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6876.499407] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6884.214803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6884.591281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6892.289743] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6901.976975] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6906.104952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6917.262197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6917.373421] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6917.811623] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6926.171198] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6935.013680] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6936.821670] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6949.226147] Lustre: Permanently deactivating lustre-OST0001 [ 6949.254406] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6955.608550] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6962.591469] Lustre: Permanently reactivating lustre-OST0001 [ 6962.620159] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6962.625499] Lustre: Skipped 2 previous similar messages [ 6963.528721] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6963.566691] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6963.585132] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6963.612329] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6963.660375] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6975.469654] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6975.475811] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6975.492458] Lustre: Skipped 2 previous similar messages [ 6975.520281] Lustre: Skipped 1 previous similar message [ 6980.576903] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6980.590183] Lustre: Skipped 1 previous similar message [ 6981.638101] Lustre: server umount lustre-OST0001 complete [ 6985.730919] LustreError: 220822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6985.757369] LustreError: 220822:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6988.260897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6988.271062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.302609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6990.306166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6990.314471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6990.336234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6990.818831] LustreError: 220821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6990.829516] LustreError: 220821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6993.004579] Lustre: server umount lustre-OST0000 complete [ 7001.058211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7001.064074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7001.090250] Lustre: Skipped 1 previous similar message [ 7001.097866] Lustre: Skipped 1 previous similar message [ 7004.062324] Lustre: server umount lustre-MDT0000 complete [ 7006.178136] LustreError: 218997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7006.196622] LustreError: 218997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7008.654183] LustreError: 218979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783975389 with bad export cookie 11837341233252624520 [ 7008.666885] LustreError: 218979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7008.667692] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7009.028303] Lustre: server umount lustre-MDT0001 complete [ 7022.400382] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 7025.798530] Key type lgssc unregistered [ 7026.191573] LNet: 224064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7026.197767] LNetError: 224064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7027.241418] LNet: Removed LNI 192.168.204.116@tcp [ 7028.337701] Key type .llcrypt unregistered [ 7028.340349] Key type ._llcrypt unregistered [ 7073.944718] Key type ._llcrypt registered [ 7073.948802] Key type .llcrypt registered [ 7074.141507] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7075.571407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7075.615046] alg: No test for adler32 (adler32-zlib) [ 7076.722803] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 7077.194445] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 7078.927234] Key type lgssc registered [ 7080.578414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7089.873965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7089.893525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7091.156404] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7091.176648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7091.357421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7096.165669] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7106.434340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7106.547861] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7106.573454] Lustre: 226274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7106.848483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7112.336436] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7120.494124] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7126.230693] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7133.137847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7133.355837] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7133.650473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7139.488964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7140.321545] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7148.041932] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7152.313573] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7159.255145] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7166.068527] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7172.105553] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7181.174515] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7203.091237] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7203.097281] Lustre: Skipped 2 previous similar messages [ 7238.668623] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7238.673306] Lustre: Skipped 4 previous similar messages [ 7309.839264] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7309.850285] Lustre: Skipped 8 previous similar messages [ 7319.008479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7319.027954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.032864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7319.044616] Lustre: Skipped 1 previous similar message [ 7324.135155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7324.144484] Lustre: Skipped 2 previous similar messages [ 7324.890878] Lustre: server umount lustre-OST0000 complete [ 7329.252804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7329.288047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7329.301540] Lustre: Skipped 1 previous similar message [ 7334.371854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7334.382979] Lustre: Skipped 1 previous similar message [ 7335.182032] Lustre: server umount lustre-MDT0000 complete [ 7339.212958] LustreError: 229505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783975720 with bad export cookie 6154032732879554840 [ 7339.230155] LustreError: 229505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7339.237032] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7339.549918] Lustre: server umount lustre-MDT0001 complete [ 7352.249849] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 7356.359880] Key type lgssc unregistered [ 7356.832905] LNet: 231059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7356.861437] LNetError: 231059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7356.880172] LNet: Removed LNI 192.168.204.116@tcp [ 7357.877332] Key type .llcrypt unregistered [ 7357.881462] Key type ._llcrypt unregistered [ 7383.169181] Key type ._llcrypt registered [ 7383.174080] Key type .llcrypt registered [ 7383.365996] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7385.297976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7385.760789] alg: No test for adler32 (adler32-zlib) [ 7386.972742] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 7387.260686] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 7388.951322] Key type lgssc registered [ 7390.396818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7401.685421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7401.727135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7403.223475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7408.664797] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7413.732603] LustreError: 232476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7413.748165] LustreError: 232476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7418.850684] LustreError: 232477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7418.925529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7419.710978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7425.745598] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7435.170550] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7442.423898] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7450.499049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7450.963915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7457.893160] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7467.388405] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7474.610831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7505.389349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7505.395896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7505.407555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7505.424375] Lustre: Skipped 1 previous similar message [ 7505.448769] Lustre: Skipped 1 previous similar message [ 7510.032828] Lustre: server umount lustre-OST0000 complete [ 7515.427039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7515.444422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7515.451137] Lustre: Skipped 1 previous similar message [ 7520.736688] LustreError: 232471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7520.751581] LustreError: 232471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7520.877996] Lustre: server umount lustre-MDT0000 complete [ 7525.432948] LustreError: 232457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783975906 with bad export cookie 16099669058873660590 [ 7525.438614] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7525.449523] LustreError: 232457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7525.967446] Lustre: server umount lustre-MDT0001 complete [ 7541.664667] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 7545.090550] Key type lgssc unregistered [ 7545.537907] LNet: 236218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7545.547502] LNetError: 236218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7545.578956] LNet: Removed LNI 192.168.204.116@tcp [ 7546.661406] Key type .llcrypt unregistered [ 7546.670121] Key type ._llcrypt unregistered [ 7574.551716] Key type ._llcrypt registered [ 7574.553641] Key type .llcrypt registered [ 7574.687917] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7576.483762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7576.512948] alg: No test for adler32 (adler32-zlib) [ 7577.696241] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 7577.961235] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 7579.672872] Key type lgssc registered [ 7581.391586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7853.784777] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 7856.653123] Key type lgssc unregistered [ 7857.043976] LNet: 238240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7857.056541] LNetError: 238240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7857.072615] LNet: Removed LNI 192.168.204.116@tcp [ 7858.116129] Key type .llcrypt unregistered [ 7858.117941] Key type ._llcrypt unregistered [ 7888.396561] Key type ._llcrypt registered [ 7888.399617] Key type .llcrypt registered [ 7888.522349] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 7907.284877] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7908.573807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7908.587897] alg: No test for adler32 (adler32-zlib) [ 7909.815748] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 7910.200887] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 7911.959902] Key type lgssc registered [ 7913.041877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7946.401698] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7667 sec ========== 16:58:44 (1783976324) [ 7948.643911] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:58:46 (1783976326) === [ 7952.937048] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:58:51 (1783976331) === [ 7999.977929] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 8003.033329] Key type lgssc unregistered [ 8003.350880] LNet: 243482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8003.363562] LNetError: 243482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8003.372806] LNet: Removed LNI 192.168.204.116@tcp [ 8004.373677] Key type .llcrypt unregistered [ 8004.385770] Key type ._llcrypt unregistered