[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483381297 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003368] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005020] kvm-guest: setup PV IPIs [ 0.008750] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010022] pid_max: default: 32768 minimum: 301 [ 0.011162] LSM: Security Framework initializing [ 0.012061] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014118] *** VALIDATE selinux *** [ 0.023171] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025333] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026185] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028140] *** VALIDATE tmpfs *** [ 0.029451] *** VALIDATE proc *** [ 0.030301] *** VALIDATE cgroup *** [ 0.031015] *** VALIDATE cgroup2 *** [ 0.033059] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036045] Spectre V2 : User space: Vulnerable [ 0.037017] Speculative Store Bypass: Vulnerable [ 0.040576] debug: unmapping init [mem 0xffffffff85859000-0xffffffff85860fff] [ 0.042199] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043810] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044049] ... version: 2 [ 0.045019] ... bit width: 48 [ 0.046029] ... generic registers: 4 [ 0.047021] ... value mask: 0000ffffffffffff [ 0.048024] ... max period: 00007fffffffffff [ 0.049023] ... fixed-purpose events: 3 [ 0.050022] ... event mask: 000000070000000f [ 0.051365] rcu: Hierarchical SRCU implementation. [ 0.053594] smp: Bringing up secondary CPUs ... [ 0.054683] x86: Booting SMP configuration: [ 0.055035] .... node #0, CPUs: #1 #2 #3 [ 0.061363] smp: Brought up 1 node, 4 CPUs [ 0.063032] smpboot: Max logical packages: 1 [ 0.064022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.219000] node 0 deferred pages initialised in 151ms [ 0.222135] devtmpfs: initialized [ 0.223304] x86/mm: Memory block size: 128MB [ 0.226749] gcov: version magic: 0x41383552 [ 0.229232] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.233099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.236393] pinctrl core: initialized pinctrl subsystem [ 0.238186] [ 0.238905] ************************************************************* [ 0.241015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.244017] ** ** [ 0.246012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.249012] ** ** [ 0.251012] ** This means that this kernel is built to expose internal ** [ 0.254015] ** IOMMU data structures, which may compromise security on ** [ 0.256013] ** your system. ** [ 0.259017] ** ** [ 0.261012] ** If you see this message and you are not debugging the ** [ 0.264016] ** kernel, report this immediately to your vendor! ** [ 0.266012] ** ** [ 0.268016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.270014] ************************************************************* [ 0.272671] NET: Registered protocol family 16 [ 0.274318] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.276088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.279076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.283157] cpuidle: using governor menu [ 0.285117] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.286729] PCI: Using configuration type 1 for base access [ 0.289134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.300157] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.303058] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.307038] cryptd: max_cpu_qlen set to 1000 [ 0.313306] ACPI: Added _OSI(Module Device) [ 0.315028] ACPI: Added _OSI(Processor Device) [ 0.317024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.318020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.323238] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.329631] ACPI: Interpreter enabled [ 0.330077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.332016] ACPI: Using IOAPIC for interrupt routing [ 0.334122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.339716] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.350830] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.354058] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.356019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.359093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.364430] acpiphp: Slot [2] registered [ 0.366145] acpiphp: Slot [5] registered [ 0.367130] acpiphp: Slot [6] registered [ 0.369317] acpiphp: Slot [7] registered [ 0.371149] acpiphp: Slot [8] registered [ 0.374209] acpiphp: Slot [9] registered [ 0.376136] acpiphp: Slot [10] registered [ 0.377100] acpiphp: Slot [3] registered [ 0.379097] acpiphp: Slot [4] registered [ 0.380172] acpiphp: Slot [11] registered [ 0.381125] acpiphp: Slot [12] registered [ 0.382089] acpiphp: Slot [13] registered [ 0.383072] acpiphp: Slot [14] registered [ 0.385107] acpiphp: Slot [15] registered [ 0.387112] acpiphp: Slot [16] registered [ 0.388081] acpiphp: Slot [17] registered [ 0.389080] acpiphp: Slot [18] registered [ 0.390124] acpiphp: Slot [19] registered [ 0.391090] acpiphp: Slot [20] registered [ 0.392083] acpiphp: Slot [21] registered [ 0.396154] acpiphp: Slot [22] registered [ 0.397095] acpiphp: Slot [23] registered [ 0.398089] acpiphp: Slot [24] registered [ 0.400122] acpiphp: Slot [25] registered [ 0.401097] acpiphp: Slot [26] registered [ 0.402113] acpiphp: Slot [27] registered [ 0.403087] acpiphp: Slot [28] registered [ 0.404080] acpiphp: Slot [29] registered [ 0.405102] acpiphp: Slot [30] registered [ 0.406087] acpiphp: Slot [31] registered [ 0.407056] PCI host bridge to bus 0000:00 [ 0.408023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.410021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.412023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.414023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.415019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.417029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.419177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.422382] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.425164] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.436015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.439582] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.442016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.444017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.446019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.449612] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.453837] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.457043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.459459] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.466027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.480000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.484774] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.490753] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.500015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.509025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.526018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.537510] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.545020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.555024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.580022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.592365] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.601024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.609021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.631022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.646955] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.660091] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.671025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.693020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.710325] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.726019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.745021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.802021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.817664] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.827017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.844022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.869020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.884716] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.889453] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.893702] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.896430] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.899316] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.903330] iommu: Default domain type: Passthrough [ 0.904000] SCSI subsystem initialized [ 0.905162] ACPI: bus type USB registered [ 0.906117] usbcore: registered new interface driver usbfs [ 0.909107] usbcore: registered new interface driver hub [ 0.910094] usbcore: registered new device driver usb [ 0.911000] pps_core: LinuxPPS API ver. 1 registered [ 0.913024] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.916102] PTP clock support registered [ 0.919070] EDAC MC: Ver: 3.0.0 [ 0.921152] PCI: Using ACPI for IRQ routing [ 0.924095] NetLabel: Initializing [ 0.925017] NetLabel: domain hash size = 128 [ 0.927032] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.930107] NetLabel: unlabeled traffic allowed by default [ 0.932193] vgaarb: loaded [ 0.934289] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.936013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.940772] clocksource: Switched to clocksource kvm-clock [ 1.092518] VFS: Disk quotas dquot_6.6.0 [ 1.095824] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.100591] *** VALIDATE ramfs *** [ 1.102262] *** VALIDATE hugetlbfs *** [ 1.104259] pnp: PnP ACPI init [ 1.107074] pnp: PnP ACPI: found 6 devices [ 1.130595] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.135090] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.137809] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.140665] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.143334] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.146309] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.149942] NET: Registered protocol family 2 [ 1.152547] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.158281] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.163197] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.172410] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.176788] TCP: Hash tables configured (established 65536 bind 65536) [ 1.180166] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.183156] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.187116] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.192179] NET: Registered protocol family 1 [ 1.197403] RPC: Registered named UNIX socket transport module. [ 1.199666] RPC: Registered udp transport module. [ 1.203067] RPC: Registered tcp transport module. [ 1.205229] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.208749] NET: Registered protocol family 44 [ 1.211393] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.214142] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.217310] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.220632] PCI: CLS 0 bytes, default 64 [ 1.223098] Unpacking initramfs... [ 2.863582] debug: unmapping init [mem 0xffff90b73cc54000-0xffff90b73ffbffff] [ 2.868816] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.870891] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.873782] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.535681] Initialise system trusted keyrings [ 3.537698] Key type blacklist registered [ 3.539861] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.549188] zbud: loaded [ 3.552265] *** VALIDATE nfs *** [ 3.553506] *** VALIDATE nfs4 *** [ 3.555253] pstore: using deflate compression [ 3.559659] Platform Keyring initialized [ 3.689618] NET: Registered protocol family 38 [ 3.691793] Key type asymmetric registered [ 3.693774] Asymmetric key parser 'x509' registered [ 3.695617] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.699562] io scheduler mq-deadline registered [ 3.701883] io scheduler kyber registered [ 3.704102] io scheduler bfq registered [ 3.706562] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.710215] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.713156] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.716434] ACPI: Power Button [PWRF] [ 3.723728] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.731439] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.766133] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.781275] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.812940] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.846904] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.878717] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.884377] Non-volatile memory driver v1.3 [ 3.886465] Linux agpgart interface v0.103 [ 3.936585] virtio_blk virtio1: [vda] 150040 512-byte logical blocks (76.8 MB/73.3 MiB) [ 3.939920] vda: detected capacity change from 0 to 76820480 [ 3.971557] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.976157] vdb: detected capacity change from 0 to 1073741824 [ 3.999351] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.002551] vdc: detected capacity change from 0 to 2621440000 [ 4.032319] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.036323] vdd: detected capacity change from 0 to 2621440000 [ 4.057442] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.062160] vde: detected capacity change from 0 to 4294967296 [ 4.096175] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.100835] vdf: detected capacity change from 0 to 4294967296 [ 4.113708] libphy: Fixed MDIO Bus: probed [ 4.123262] usbcore: registered new interface driver usbserial_generic [ 4.127461] usbserial: USB Serial support registered for generic [ 4.132158] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.142462] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.144476] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.148660] mousedev: PS/2 mouse device common for all mice [ 4.156518] rtc_cmos 00:05: RTC can wake from S4 [ 4.160807] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.164728] rtc_cmos 00:05: registered as rtc0 [ 4.170786] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.173293] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.173344] intel_pstate: CPU model not supported [ 4.180801] hid: raw HID events driver (C) Jiri Kosina [ 4.189415] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.196500] usbcore: registered new interface driver usbhid [ 4.196508] usbhid: USB HID core driver [ 4.196619] drop_monitor: Initializing network drop monitor service [ 4.196769] Initializing XFRM netlink socket [ 4.197198] NET: Registered protocol family 10 [ 4.199944] Segment Routing with IPv6 [ 4.219458] NET: Registered protocol family 17 [ 4.222522] mpls_gso: MPLS GSO support [ 4.229912] RAS: Correctable Errors collector initialized. [ 4.232222] AVX version of gcm_enc/dec engaged. [ 4.235354] AES CTR mode by8 optimization enabled [ 4.324923] sched_clock: Marking stable (4324758992, 0)->(5344309375, -1019550383) [ 4.329299] registered taskstats version 1 [ 4.331919] Loading compiled-in X.509 certificates [ 4.333449] zswap: loaded using pool lzo/zbud [ 4.362984] Key type big_key registered [ 4.375940] Key type encrypted registered [ 4.378026] ima: No TPM chip found, activating TPM-bypass! [ 4.380188] ima: Allocated hash algorithm: sha1 [ 4.382065] ima: No architecture policies found [ 4.384251] evm: Initialising EVM extended attributes: [ 4.388668] evm: security.selinux [ 4.390016] evm: security.ima [ 4.391074] evm: security.capability [ 4.392445] evm: HMAC attrs: 0x1 [ 4.395054] rtc_cmos 00:05: setting system clock to 2026-09-07 01:52:03 UTC (1788745923) [ 4.403125] debug: unmapping init [mem 0xffffffff86803000-0xffffffff869fffff] [ 4.406645] debug: unmapping init [mem 0xffffffff85582000-0xffffffff85858fff] [ 4.416067] Write protecting the kernel read-only data: 28672k [ 4.420355] debug: unmapping init [mem 0xffffffff83c03000-0xffffffff83dfffff] [ 4.424345] debug: unmapping init [mem 0xffffffff84514000-0xffffffff845fffff] [ 4.472651] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.484173] systemd[1]: Detected virtualization kvm. [ 4.486205] systemd[1]: Detected architecture x86-64. [ 4.489333] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.524737] systemd[1]: No hostname configured. [ 4.526820] systemd[1]: Set hostname to . [ 4.529394] random: systemd: uninitialized urandom read (16 bytes read) [ 4.532389] systemd[1]: Initializing machine ID from random generator. [ 4.580468] random: ln: uninitialized urandom read (6 bytes read) [ 4.723271] random: systemd: uninitialized urandom read (16 bytes read) [ 4.727281] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.736334] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.740803] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.650232] device-mapper: uevent: version 1.0.3 [ 5.652940] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.690428] scsi host0: ata_piix [ 6.747566] virtio_net virtio0 ens2: renamed from eth0 [ 6.780298] random: fast init done [ 6.783462] scsi host1: ata_piix [ 6.786623] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.790508] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.918534] random: crng init done [ 11.922505] random: 7 urandom warning(s) missed due to ratelimiting [ 13.179368] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.122166] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.836814] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.196278] SELinux: Disabled at runtime. [ 16.273223] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.283432] systemd[1]: Detected virtualization kvm. [ 16.291270] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.076281] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.080523] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.092389] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.099273] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.105044] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.114836] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.119877] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 17.327457] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 18.330478] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.143399] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.295780] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.823596] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.969320] EDAC sbridge: Ver: 1.1.2 [ 23.426770] Key type dns_resolver registered [ 23.858926] NFS: Registering the id_resolver key type [ 23.861244] Key type id_resolver registered [ 23.862491] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg651-server login: [ 61.918197] hrtimer: interrupt took 8186148 ns [ 82.344602] libcfs: loading out-of-tree module taints kernel. [ 82.405620] Key type ._llcrypt registered [ 82.407253] Key type .llcrypt registered [ 82.523044] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 108.033259] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 109.892928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 109.917515] alg: No test for adler32 (adler32-zlib) [ 111.534739] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 112.428347] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 114.168448] Key type lgssc registered [ 115.974992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 138.481331] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 183.597780] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 197.618182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 197.662388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.018873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 199.080571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 199.211347] Lustre: lustre-MDT0000: new disk, initializing [ 199.367161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.396833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.918888] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.341785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.423882] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.479217] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 220.484173] Lustre: Skipped 1 previous similar message [ 220.622209] Lustre: lustre-MDT0001: new disk, initializing [ 220.693739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.732541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 220.755885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.642194] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.919901] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 241.832884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.229041] Lustre: lustre-OST0000: new disk, initializing [ 242.237364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 242.247234] Lustre: 8460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.310222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.630076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.645274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 242.731885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 248.813611] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.421990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 265.584192] Lustre: lustre-OST0001: new disk, initializing [ 265.587249] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 265.591510] Lustre: 9533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 265.672930] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.598397] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.474200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 273.489903] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 273.575758] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 285.440084] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.809554] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 302.467858] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing check_logdir /tmp/testlogs/ [ 308.304702] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing yml_node [ 313.640882] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 316.584549] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 319.892856] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 321.615183] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 21:57:20 EDT 2026 [ 342.619610] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 344.475878] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 360.940902] LustreError: 3650:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff90b683d15f80 x1875636158300672/t0(0) o41->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-pre-1-0.0' uid:0 gid:0 projid:4294967295 [ 363.999515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 364.010361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.020594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.048609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.052405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.085946] Lustre: Skipped 1 previous similar message [ 366.093808] Lustre: Skipped 1 previous similar message [ 371.185398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 371.201410] Lustre: Skipped 5 previous similar messages [ 375.263299] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 375.721237] Lustre: server umount lustre-MDT0000 complete [ 381.409723] LustreError: 7837:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 381.444736] LustreError: 7837:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 8 previous similar messages [ 381.738537] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746300 with bad export cookie 8710085468337653162 [ 381.747586] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.758238] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 386.529537] LustreError: 7731:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.535989] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.565731] LustreError: 7731:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 386.598797] Lustre: Skipped 1 previous similar message [ 386.641664] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 386.659932] Lustre: Skipped 1 previous similar message [ 391.650287] LustreError: 6527:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 391.651978] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 391.676807] LustreError: 6527:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 391.691318] Lustre: Skipped 1 previous similar message [ 396.255131] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 396.698410] Lustre: server umount lustre-MDT0001 complete [ 401.419497] Lustre: server umount lustre-OST0000 complete [ 406.345385] Lustre: server umount lustre-OST0001 complete [ 414.490085] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 426.266925] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 470.296607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.537680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 470.587633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 470.708337] Lustre: lustre-MDT0000: new disk, initializing [ 470.880039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.912421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 475.831295] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 488.094783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.202639] Lustre: 15093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 488.248987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 488.255200] Lustre: Skipped 1 previous similar message [ 488.356452] Lustre: lustre-MDT0001: new disk, initializing [ 488.474040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 488.540721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 488.550746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 493.337963] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 504.368713] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 510.327167] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 517.359655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 517.619723] Lustre: lustre-OST0000: new disk, initializing [ 517.625489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 517.631682] Lustre: 17061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 517.733788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 518.755843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 518.767401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 518.911669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 523.721857] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 534.643643] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 539.867250] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 540.023633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 545.593621] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 545.798846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 549.856383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 549.860623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.862917] LustreError: Skipped 1 previous similar message [ 549.864532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 549.864541] Lustre: Skipped 2 previous similar messages [ 549.880958] Lustre: Skipped 2 previous similar messages [ 562.143254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 562.358088] Lustre: server umount lustre-OST0000 complete [ 570.336274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 570.345049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.346742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 570.356920] Lustre: Skipped 1 previous similar message [ 570.372524] Lustre: Skipped 6 previous similar messages [ 572.459349] Lustre: server umount lustre-MDT0000 complete [ 575.461256] LustreError: 15977:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.485545] LustreError: 15977:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 576.305541] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746495 with bad export cookie 8710085468337654695 [ 576.319804] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 576.320819] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 576.703239] Lustre: server umount lustre-MDT0001 complete [ 581.070749] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:01:39 (1788746499) [ 583.652278] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 585.370952] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:01:44 (1788746504) [ 591.840731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 592.225045] LustreError: 19485:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.305042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 597.190249] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 602.598894] LustreError: 19486:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.618974] LustreError: 19486:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 606.268917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.787771] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 617.717240] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 623.272725] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 630.443057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 637.175800] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 645.172349] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 657.378626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 657.385843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.407153] LustreError: Skipped 1 previous similar message [ 657.409412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 657.448042] Lustre: Skipped 1 previous similar message [ 659.899237] Lustre: server umount lustre-OST0000 complete [ 671.694937] Lustre: server umount lustre-MDT0000 complete [ 672.736917] LustreError: 19486:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.757778] LustreError: 19486:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 675.094052] LustreError: 19467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746594 with bad export cookie 8710085468337655577 [ 675.108582] LustreError: 19467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 675.112684] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 675.479856] Lustre: server umount lustre-MDT0001 complete [ 685.508379] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 687.932915] Key type lgssc unregistered [ 688.196548] LNet: 23019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.204850] LNetError: 23019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.223986] LNet: Removed LNI 192.168.206.151@tcp [ 689.104407] Key type .llcrypt unregistered [ 689.108914] Key type ._llcrypt unregistered [ 710.490022] Key type ._llcrypt registered [ 710.497496] Key type .llcrypt registered [ 710.633948] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 711.575149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 711.715473] alg: No test for adler32 (adler32-zlib) [ 712.763947] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 713.018050] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 714.704780] Key type lgssc registered [ 715.902509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 724.333495] loop: module loaded [ 724.392174] loop0: detected capacity change from 0 to 204800000 [ 724.610412] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 724.627810] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 724.978200] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 730.223563] loop0: detected capacity change from 0 to 204800000 [ 730.407900] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 730.428584] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 730.444474] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 734.669714] loop0: detected capacity change from 0 to 204800000 [ 739.627869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 739.671723] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 740.843835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 740.903923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 740.982552] Lustre: lustre-MDT0000: new disk, initializing [ 741.061691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 741.078490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 745.165429] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 753.324987] loop1: detected capacity change from 0 to 204800000 [ 758.514936] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 758.831448] Lustre: lustre-OST07c6: new disk, initializing [ 758.835873] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 758.844686] Lustre: Skipped 1 previous similar message [ 758.855721] Lustre: 26202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 758.937612] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 764.294339] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 765.524337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 765.530879] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 765.565435] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 768.314979] Lustre: 25189:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 768.322060] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 768.338457] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 768.353644] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 768.358663] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 768.368770] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 781.651810] Lustre: 25190:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 781.666857] Lustre: 25190:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 781.673403] Lustre: 25190:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 781.689980] Lustre: 25190:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 781.699459] Lustre: 25190:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 781.705288] Lustre: 25190:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 786.521498] Lustre: 25190:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 786.543187] Lustre: 25190:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 786.549580] Lustre: 25190:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 786.555907] Lustre: 25190:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.568632] Lustre: 25190:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 786.581386] Lustre: 25190:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.589318] Lustre: 25190:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 786.596598] Lustre: 25190:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.610838] Lustre: 25190:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 786.617328] Lustre: 25190:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.621304] Lustre: 25190:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 786.625550] Lustre: 25190:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.817218] Lustre: 25189:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 789.826363] Lustre: 25189:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 789.838300] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 789.854174] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.865306] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 789.872682] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.879338] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 789.884787] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.890616] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 789.897447] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.909243] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 789.919312] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.546446] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 794.558979] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 794.569271] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 794.576053] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.581070] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 794.587461] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.595297] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 794.603990] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.612235] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 794.618789] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.624578] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 794.642557] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.767364] Lustre: 25190:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 805.786712] Lustre: 25190:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 805.795351] Lustre: 25190:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 805.806981] Lustre: 25190:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 805.814846] Lustre: 25190:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 805.818793] Lustre: 25190:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 805.822742] Lustre: 25190:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 805.827097] Lustre: 25190:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 805.831616] Lustre: 25190:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 805.835049] Lustre: 25190:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 805.838908] Lustre: 25190:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 805.842365] Lustre: 25190:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 852.451831] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.454974] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 852.469673] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 857.570486] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 858.438892] Lustre: server umount lustre-OST07c6 complete [ 866.486495] Lustre: server umount lustre-MDT0000 complete [ 880.663865] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 882.867508] Key type lgssc unregistered [ 883.129675] LNet: 28758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.135736] LNetError: 28758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.151377] LNet: Removed LNI 192.168.206.151@tcp [ 883.842216] Key type .llcrypt unregistered [ 883.844434] Key type ._llcrypt unregistered [ 899.393748] Key type ._llcrypt registered [ 899.398183] Key type .llcrypt registered [ 899.513541] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 900.220621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 900.280113] alg: No test for adler32 (adler32-zlib) [ 901.340565] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 901.557579] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 903.255852] Key type lgssc registered [ 904.141477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 910.146799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 910.167167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.486470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 914.860286] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 921.060970] LustreError: 30005:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 921.078256] LustreError: 30005:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 921.633099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 921.893226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 925.626986] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 931.681484] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 936.172388] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 941.359289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 941.693817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.752990] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 953.008546] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 956.016226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 971.751627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 971.759130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.766102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 972.768402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 972.776449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.788365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 974.304235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 974.313278] Lustre: Skipped 1 previous similar message [ 976.058622] Lustre: server umount lustre-OST0000 complete [ 979.424541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.426231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.432384] Lustre: Skipped 1 previous similar message [ 984.555889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 984.566554] Lustre: Skipped 2 previous similar messages [ 993.247208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 993.398118] Lustre: server umount lustre-MDT0000 complete [ 994.787249] LustreError: 32462:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 994.805336] LustreError: 32462:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 996.478355] LustreError: 29985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788746915 with bad export cookie 15973826430488254993 [ 996.483054] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 996.493583] LustreError: 29985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 996.822174] Lustre: server umount lustre-MDT0001 complete [ 1005.356339] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1007.509740] Key type lgssc unregistered [ 1007.777468] LNet: 33738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.784175] LNetError: 33738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.800363] LNet: Removed LNI 192.168.206.151@tcp [ 1008.505623] Key type .llcrypt unregistered [ 1008.507662] Key type ._llcrypt unregistered [ 1022.479259] Key type ._llcrypt registered [ 1022.481970] Key type .llcrypt registered [ 1022.558518] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1023.258966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1023.275872] alg: No test for adler32 (adler32-zlib) [ 1024.367877] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 1024.569861] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1026.224130] Key type lgssc registered [ 1027.193509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1033.213777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1033.236981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1034.653502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1038.379943] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1043.940547] LustreError: 34984:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1043.958561] LustreError: 34984:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1044.806570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.067719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1048.478545] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1053.862771] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1057.797699] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1062.801808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1063.084411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.067835] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1073.984097] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1079.477365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1090.018695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1090.025471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.030881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1090.037754] Lustre: Skipped 1 previous similar message [ 1093.360636] Lustre: server umount lustre-OST0000 complete [ 1096.162996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1096.174632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.190740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.194771] Lustre: Skipped 1 previous similar message [ 1100.258124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1100.260225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.423954] Lustre: server umount lustre-MDT0000 complete [ 1104.867578] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747023 with bad export cookie 9392319743111755948 [ 1104.875472] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1104.878920] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1105.103236] Lustre: server umount lustre-MDT0001 complete [ 1112.028752] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1113.953978] Key type lgssc unregistered [ 1114.157440] LNet: 38534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.166229] LNetError: 38534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.183160] LNet: Removed LNI 192.168.206.151@tcp [ 1114.801166] Key type .llcrypt unregistered [ 1114.803900] Key type ._llcrypt unregistered [ 1129.995559] Key type ._llcrypt registered [ 1129.997613] Key type .llcrypt registered [ 1130.066628] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1130.604788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1130.616393] alg: No test for adler32 (adler32-zlib) [ 1131.600253] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 1131.775392] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1133.423300] Key type lgssc registered [ 1134.439249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1140.638683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1140.656427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1142.026155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.013867] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1151.047896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.336353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1154.457132] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1160.337169] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1164.651725] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1170.544115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1170.866483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1175.569901] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1180.702263] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1187.247471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1196.784493] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 1197.542249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1197.545715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1197.546707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1197.553127] Lustre: Skipped 1 previous similar message [ 1197.896391] Lustre: server umount lustre-MDT0000 complete [ 1200.610145] LustreError: 40660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747119 with bad export cookie 13574228824240710936 [ 1200.610681] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1200.617659] LustreError: 40660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1201.871256] LustreError: 39923:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.890045] LustreError: 39923:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1201.892212] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 1201.898382] Lustre: Skipped 2 previous similar messages [ 1202.659454] LustreError: 39922:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.659832] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.676979] Lustre: Skipped 1 previous similar message [ 1202.922232] Lustre: server umount lustre-MDT0001 complete [ 1214.747853] LustreError: 39908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.51@tcp arrived at 1788747133 with bad export cookie 13574228824240711573 [ 1214.758397] LustreError: 39908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1222.607627] Lustre: server umount lustre-OST0000 complete [ 1233.770937] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1235.726726] Key type lgssc unregistered [ 1235.941689] LNet: 43803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.951850] LNetError: 43803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1235.964632] LNet: Removed LNI 192.168.206.151@tcp [ 1236.590892] Key type .llcrypt unregistered [ 1236.592955] Key type ._llcrypt unregistered [ 1253.540268] Key type ._llcrypt registered [ 1253.545770] Key type .llcrypt registered [ 1253.630155] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1254.248259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1254.320852] alg: No test for adler32 (adler32-zlib) [ 1255.346465] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 1255.504151] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1257.136694] Key type lgssc registered [ 1257.974435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1263.060858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1263.071787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.361242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.160207] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1273.209937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.502430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1276.350540] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1280.793831] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1284.241522] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1288.869132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1289.086902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.160187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1293.229768] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1298.123352] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1303.522898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.524261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1303.535228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1304.795597] Lustre: lustre-OST0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 1304.802588] Lustre: Skipped 1 previous similar message [ 1308.642765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1308.792209] Lustre: server umount lustre-OST0000 complete [ 1325.026261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1325.033090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.042454] Lustre: Skipped 1 previous similar message [ 1325.046080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.049873] Lustre: Skipped 1 previous similar message [ 1329.125252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1329.128566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1330.785258] Lustre: server umount lustre-MDT0000 complete [ 1333.280961] LustreError: 45919:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1333.295357] LustreError: 45919:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 1333.849174] LustreError: 45173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747252 with bad export cookie 14594506693804210778 [ 1333.851146] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1333.858788] LustreError: 45173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1340.208313] Lustre: server umount lustre-MDT0001 complete [ 1348.150247] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1350.150212] Key type lgssc unregistered [ 1350.411661] LNet: 48894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1350.419335] LNetError: 48894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1350.439778] LNet: Removed LNI 192.168.206.151@tcp [ 1351.091660] Key type .llcrypt unregistered [ 1351.093775] Key type ._llcrypt unregistered [ 1368.652579] Key type ._llcrypt registered [ 1368.654227] Key type .llcrypt registered [ 1368.766221] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1369.451994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1369.492166] alg: No test for adler32 (adler32-zlib) [ 1370.460153] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 1370.665410] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1372.303207] Key type lgssc registered [ 1373.157747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1378.854333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1378.870314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.167150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.377713] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1389.558321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.824665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1393.513414] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1399.273503] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1403.211960] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1408.156109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1408.409974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.015793] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1418.595685] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1425.840020] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1432.620716] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1435.176602] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1446.472110] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0622h16m05s [ 1447.430573] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1449.686740] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 1450.701464] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 1451.491800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1451.492400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.510446] Lustre: Skipped 2 previous similar messages [ 1452.580400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1452.587580] Lustre: Skipped 5 previous similar messages [ 1454.845330] Lustre: server umount lustre-MDT0000 complete [ 1455.755564] LustreError: 50284:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.769484] LustreError: 50284:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1457.464426] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1458.657191] LustreError: 50288:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1458.668429] LustreError: 50288:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 1461.964989] LustreError: 50283:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1462.188386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1462.291555] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1462.468418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1462.523505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1463.464169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1465.707411] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1468.652990] Lustre: DEBUG MARKER: Wait for df (36707) ... [ 1470.010449] Lustre: DEBUG MARKER: done [ 1474.021154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.022621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1474.034031] Lustre: Skipped 1 previous similar message [ 1474.046664] Lustre: Skipped 1 previous similar message [ 1479.138830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1479.141660] Lustre: Skipped 1 previous similar message [ 1479.806539] Lustre: server umount lustre-OST0000 complete [ 1484.259078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1484.259816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1484.272374] Lustre: Skipped 1 previous similar message [ 1489.119797] Lustre: server umount lustre-MDT0000 complete [ 1490.401773] LustreError: 50288:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.424333] LustreError: 50288:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1492.536716] LustreError: 50268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747411 with bad export cookie 7701690383229581854 [ 1492.541118] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1492.556623] LustreError: 50268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1492.952599] Lustre: server umount lustre-MDT0001 complete [ 1502.896731] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1505.221619] Key type lgssc unregistered [ 1505.484922] LNet: 55227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1505.494538] LNetError: 55227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1505.508141] LNet: Removed LNI 192.168.206.151@tcp [ 1506.304254] Key type .llcrypt unregistered [ 1506.305853] Key type ._llcrypt unregistered [ 1544.139536] Key type ._llcrypt registered [ 1544.142411] Key type .llcrypt registered [ 1544.271424] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1545.269100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1545.390509] alg: No test for adler32 (adler32-zlib) [ 1546.458948] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 1546.651310] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1548.274344] Key type lgssc registered [ 1549.283544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1556.168024] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1556.196235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1557.520762] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1557.558750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1557.779243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.608458] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1569.099389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.182029] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1569.204892] Lustre: 57429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1569.428249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1573.296311] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1578.843862] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1583.135498] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1588.560546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1588.736293] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1588.928154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1594.203725] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1598.036224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1600.486659] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1605.972299] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0622h18m44s [ 1607.205188] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1613.377409] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1615.578798] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1615.719161] LustreError: 59959:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1636.077419] Lustre: lustre-MDT0000: Client f2acf2b4-587e-47e8-943f-a27b20fb2288 (at 192.168.206.51@tcp) reconnecting [ 1655.727152] LustreError: 59959:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1657.928689] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1660.445957] Lustre: DEBUG MARKER: done [ 1669.609197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1669.610187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.624364] LustreError: Skipped 1 previous similar message [ 1669.625711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1669.653138] Lustre: Skipped 1 previous similar message [ 1671.331566] Lustre: server umount lustre-OST0000 complete [ 1674.720684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.722810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1674.744848] Lustre: Skipped 1 previous similar message [ 1674.750336] Lustre: Skipped 1 previous similar message [ 1675.808673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1675.814942] Lustre: Skipped 2 previous similar messages [ 1680.657064] Lustre: server umount lustre-MDT0000 complete [ 1683.366170] LustreError: 57420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747602 with bad export cookie 10645785763494234953 [ 1683.370363] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1683.373364] LustreError: 57420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1683.616615] Lustre: server umount lustre-MDT0001 complete [ 1690.716274] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1692.543445] Key type lgssc unregistered [ 1692.745970] LNet: 61756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.750601] LNetError: 61756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1692.765964] LNet: Removed LNI 192.168.206.151@tcp [ 1693.299908] Key type .llcrypt unregistered [ 1693.301855] Key type ._llcrypt unregistered [ 1722.797375] Key type ._llcrypt registered [ 1722.799850] Key type .llcrypt registered [ 1722.920787] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1723.746364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1723.796340] alg: No test for adler32 (adler32-zlib) [ 1724.865889] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 1725.118115] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1726.799180] Key type lgssc registered [ 1727.819489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1730.802297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1730.827394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1732.119333] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1732.141470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1732.290498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.206544] Lustre: Failing over lustre-MDT0000 [ 1733.445785] Lustre: server umount lustre-MDT0000 complete [ 1739.981894] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:20:59 (1788747659) [ 1745.622215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1745.932886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.676813] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1755.691700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1755.773112] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1755.799913] Lustre: 64530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1755.999845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1758.981206] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1764.525933] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1768.878792] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1774.598156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1774.834634] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1774.993315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.677516] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1780.842353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1785.744833] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1790.996771] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1797.088455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1797.096808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.118543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1798.289421] Lustre: server umount lustre-MDT0000 complete [ 1801.188855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.211418] Lustre: Skipped 1 previous similar message [ 1801.323400] LustreError: 64523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747720 with bad export cookie 2719348175835021518 [ 1801.327387] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1801.332882] LustreError: 64523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1801.717867] Lustre: server umount lustre-MDT0001 complete [ 1804.724836] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1808.810491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1816.612548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.688673] Lustre: 63453:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788747725/real 1788747725] req@00000000a25eb9c2 x1875637849591680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788747741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1822.712107] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1826.849962] LustreError: 63450:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a25eb9c2 x1875637849592576/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1827.094122] LustreError: 66379:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1827.109973] LustreError: 66379:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 1827.178539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.251665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1828.191241] Lustre: 63453:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788747730/real 1788747730] req@00000000b37063e8 x1875637849591808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788747746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1830.758558] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1832.223122] Lustre: 63452:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788747735/real 1788747735] req@000000001bd3bbce x1875637849592064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788747751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1832.290186] LustreError: 66380:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1832.310323] LustreError: 66380:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1837.409139] Lustre: 63452:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788747740/real 1788747740] req@000000007b9e25a5 x1875637849592320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788747756 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1837.537868] LustreError: 68299:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.565959] LustreError: 68299:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1837.668682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1838.038423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1841.139506] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1841.778882] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1847.852395] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1851.898828] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1856.481075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.483907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1856.487295] Lustre: Skipped 1 previous similar message [ 1856.496336] Lustre: Skipped 2 previous similar messages [ 1858.186851] Lustre: server umount lustre-MDT0000 complete [ 1860.536878] LustreError: 64521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747779 with bad export cookie 2719348175835023793 [ 1860.539176] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1860.546199] LustreError: 64521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1861.599771] LustreError: 68300:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1861.603207] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.620655] Lustre: Skipped 1 previous similar message [ 1861.623179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1862.841200] Lustre: server umount lustre-MDT0001 complete [ 1867.685197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1875.304064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.092171] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x25bd10c21940cf14 [ 1885.104103] Lustre: MGC192.168.206.151@tcp: Connection restored to 0@lo (at 0@lo) [ 1885.110422] Lustre: Skipped 1 previous similar message [ 1885.370721] LustreError: 66379:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.446794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1885.489793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1888.311355] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1895.904689] LustreError: 66380:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1895.937747] LustreError: 66380:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 1896.082038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.463469] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1901.083462] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1910.063451] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1910.704246] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1918.085784] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1926.118987] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.125991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1926.141658] Lustre: Skipped 2 previous similar messages [ 1926.160129] Lustre: Skipped 2 previous similar messages [ 1931.235899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1931.246335] Lustre: Skipped 2 previous similar messages [ 1931.801899] Lustre: server umount lustre-MDT0000 complete [ 1934.884489] LustreError: 64523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788747853 with bad export cookie 2719348175835025172 [ 1934.891330] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.898612] LustreError: 64523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1935.317962] Lustre: server umount lustre-MDT0001 complete [ 1941.511552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1944.495690] Lustre: DEBUG MARKER: files compared the same [ 1956.298875] Lustre: server umount lustre-OST0000 complete [ 1972.666643] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1975.617732] Key type lgssc unregistered [ 1975.833730] LNet: 74254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1975.839796] LNetError: 74254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.853833] LNet: Removed LNI 192.168.206.151@tcp [ 1976.622627] Key type .llcrypt unregistered [ 1976.629428] Key type ._llcrypt unregistered [ 1998.473443] Key type ._llcrypt registered [ 1998.476442] Key type .llcrypt registered [ 1998.589376] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1999.473852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1999.539943] alg: No test for adler32 (adler32-zlib) [ 2000.620885] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 2000.801509] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2002.487211] Key type lgssc registered [ 2003.609107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2010.703345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2010.730323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.198378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2016.936842] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug +malloc all [ 2022.373151] LustreError: 75667:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.398994] LustreError: 75667:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2023.802476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.051951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2027.770720] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug +malloc all [ 2034.144663] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2039.220278] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2045.510252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2045.713826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.840333] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug +malloc all [ 2056.000630] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2056.999222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2067.427623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2067.429082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.436856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2067.442242] Lustre: Skipped 1 previous similar message [ 2067.451571] Lustre: Skipped 1 previous similar message [ 2072.172983] Lustre: server umount lustre-OST0000 complete [ 2077.680377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.689323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2077.699308] Lustre: Skipped 1 previous similar message [ 2077.716519] Lustre: Skipped 1 previous similar message [ 2081.783948] Lustre: server umount lustre-MDT0000 complete [ 2082.789174] LustreError: 77100:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.809540] LustreError: 77100:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2085.456935] LustreError: 75647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748004 with bad export cookie 16567965952760952401 [ 2085.463169] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2085.466270] LustreError: 75647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2085.868350] Lustre: server umount lustre-MDT0001 complete [ 2094.757453] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2097.463577] Key type lgssc unregistered [ 2097.821682] LNet: 79207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.833400] LNetError: 79207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2097.858067] LNet: Removed LNI 192.168.206.151@tcp [ 2098.645388] Key type .llcrypt unregistered [ 2098.647591] Key type ._llcrypt unregistered [ 2119.068659] Key type ._llcrypt registered [ 2119.070755] Key type .llcrypt registered [ 2119.192481] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2120.224503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2120.418403] alg: No test for adler32 (adler32-zlib) [ 2121.538926] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 2121.832130] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2123.504506] Key type lgssc registered [ 2124.706862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2133.152715] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2133.171045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2148.831736] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2153.951368] LustreError: 80573:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.151@tcp: failed processing log, type 4: rc = -110 [ 2180.639768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2186.969418] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2194.803764] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2202.434864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.566204] Lustre: *** cfs_fail_loc=706, val=0*** [ 2202.956120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.005225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2206.573027] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2208.229244] LustreError: 80598:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2208.240577] LustreError: 80598:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2213.353573] LustreError: 80599:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.545335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.952852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2220.208696] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2231.844411] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2237.725085] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2240.488939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2240.502580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.514800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2241.517405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2241.533108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2246.626352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2246.633700] Lustre: Skipped 2 previous similar messages [ 2251.751447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2251.757354] Lustre: Skipped 1 previous similar message [ 2254.303724] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2254.492787] Lustre: server umount lustre-OST0000 complete [ 2260.963778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2260.972802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.998863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.320511] Lustre: server umount lustre-MDT0000 complete [ 2267.104351] LustreError: 81663:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.131941] LustreError: 81663:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2268.054803] LustreError: 80579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748187 with bad export cookie 16341103615575035160 [ 2268.060736] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2268.066458] LustreError: 80579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2268.400402] Lustre: server umount lustre-MDT0001 complete [ 2278.130847] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2281.292322] Key type lgssc unregistered [ 2281.644805] LNet: 84190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.652345] LNetError: 84190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2281.674440] LNet: Removed LNI 192.168.206.151@tcp [ 2282.522400] Key type .llcrypt unregistered [ 2282.526755] Key type ._llcrypt unregistered [ 2307.577317] Key type ._llcrypt registered [ 2307.587033] Key type .llcrypt registered [ 2307.806666] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2309.396477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2309.443269] alg: No test for adler32 (adler32-zlib) [ 2310.668455] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 2311.056316] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2312.775216] Key type lgssc registered [ 2313.855995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2323.234102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2323.247425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2329.257936] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2338.127856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.704985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2343.788169] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2343.908266] LustreError: 86292:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.938026] LustreError: 86292:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 2348.002588] LustreError: 86295:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2348.051800] LustreError: 86295:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2352.813041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2353.121820] LustreError: 86296:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.141848] LustreError: 86296:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 2353.213767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.214316] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2358.241915] LustreError: 86292:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2358.261878] LustreError: 86292:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2363.364879] LustreError: 86290:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.385068] LustreError: 86290:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2365.908039] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2365.923462] Lustre: MGS: Client ba719ca9-820e-46f8-aa45-a7620c653302 (at 0@lo) reconnecting [ 2365.939693] LustreError: 86306:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2366.143820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2366.205541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2370.229759] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2383.840409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2383.849552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2383.862319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2386.914225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2386.922252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2386.923206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2389.143368] Lustre: server umount lustre-OST0000 complete [ 2397.155837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.158157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2397.158417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2397.158421] Lustre: Skipped 1 previous similar message [ 2397.172197] Lustre: Skipped 1 previous similar message [ 2402.272694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2402.283643] Lustre: Skipped 2 previous similar messages [ 2407.391218] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2407.392956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2407.413498] Lustre: Skipped 1 previous similar message [ 2407.547310] Lustre: server umount lustre-MDT0000 complete [ 2411.332210] Lustre: server umount lustre-MDT0001 complete [ 2414.725482] Lustre: server umount MGS complete [ 2425.873493] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:32:24 (1788748344) [ 2455.235347] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 2467.032203] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2500.053667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.745461] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2512.765838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.994790] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2513.040834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2513.115365] Lustre: lustre-MDT0001: new disk, initializing [ 2513.176595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2513.210713] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2513.219654] Lustre: 93457:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2513.225453] LustreError: 93457:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2517.303117] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2526.046311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2526.290407] Lustre: lustre-OST0000: new disk, initializing [ 2526.294947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2526.325144] Lustre: 94222:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2526.379129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2531.522800] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2534.432639] Lustre: 94679:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ac7ad084 x1875638464060928/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788748469 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2540.036438] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2545.370343] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2545.380109] Lustre: MGS: Client 5d0b1d9d-f844-432c-998f-bb64eb67aca8 (at 0@lo) reconnecting [ 2545.393870] LustreError: 93443:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2545.421796] Lustre: 92725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2545.449753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2545.503856] Lustre: lustre-MDT0000: new disk, initializing [ 2545.593838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2545.619915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2545.691542] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2546.659154] Lustre: 95291:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000e39104b0 x1875638464067328/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788748481 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2549.335417] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2554.865513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2554.880358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2555.759590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2555.904292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2579.946581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2579.966596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2579.971580] Lustre: Skipped 1 previous similar message [ 2584.624586] Lustre: server umount lustre-OST0000 complete [ 2590.177206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.199363] Lustre: Skipped 2 previous similar messages [ 2594.179354] Lustre: server umount lustre-MDT0000 complete [ 2595.296694] LustreError: 93427:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.314770] LustreError: 93427:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2612.191123] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2612.448053] Lustre: server umount lustre-MDT0001 complete [ 2616.337743] Lustre: server umount MGS complete [ 2628.735052] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:35:47 (1788748547) [ 2646.315471] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2649.851610] Key type lgssc unregistered [ 2650.215209] LNet: 97790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2650.225317] LNetError: 97790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2651.241714] LNet: Removed LNI 192.168.206.151@tcp [ 2651.919736] Key type .llcrypt unregistered [ 2651.929480] Key type ._llcrypt unregistered [ 2668.205346] Key type ._llcrypt registered [ 2668.207065] Key type .llcrypt registered [ 2668.304776] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2669.285054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2669.395785] alg: No test for adler32 (adler32-zlib) [ 2670.485063] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 2670.696847] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2672.351194] Key type lgssc registered [ 2673.346061] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2679.857238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2679.873297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.210954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.136606] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2691.559975] LustreError: 99008:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.578850] LustreError: 99008:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2693.232834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2693.520288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2697.524463] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2704.375997] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2709.822783] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2715.916093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2716.276191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.283570] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2727.540497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2729.354734] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2738.148891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.153585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2738.164432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2738.174663] Lustre: Skipped 1 previous similar message [ 2743.268709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2743.293890] Lustre: Skipped 2 previous similar messages [ 2743.497799] Lustre: server umount lustre-OST0000 complete [ 2748.391520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.400695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2748.408126] Lustre: Skipped 1 previous similar message [ 2748.415293] Lustre: Skipped 1 previous similar message [ 2753.507593] LustreError: 99008:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.622924] Lustre: server umount lustre-MDT0000 complete [ 2757.848397] LustreError: 98989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748676 with bad export cookie 1582188395455477672 [ 2757.864774] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2758.131084] Lustre: server umount lustre-MDT0001 complete [ 2768.681290] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2770.891676] Key type lgssc unregistered [ 2771.151744] LNet: 102539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2771.161419] LNetError: 102539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2771.184788] LNet: Removed LNI 192.168.206.151@tcp [ 2771.947375] Key type .llcrypt unregistered [ 2771.951122] Key type ._llcrypt unregistered [ 2786.374413] Key type ._llcrypt registered [ 2786.376528] Key type .llcrypt registered [ 2786.457790] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2787.694321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2787.723325] alg: No test for adler32 (adler32-zlib) [ 2788.775264] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 2789.081480] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2790.775379] Key type lgssc registered [ 2791.901286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2797.523765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2797.527186] LustreError: 103617:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2797.549446] LustreError: 103619:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2797.559416] LustreError: 103617:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4971 [ 2797.573380] LustreError: 103617:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2797.579925] LustreError: 103617:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2797.586321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.984232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.683475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.014397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2811.954272] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2817.229516] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing lsmod [ 2819.754806] LustreError: 104823:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2819.755089] LustreError: 104824:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2819.775843] LustreError: 104823:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 2819.790753] LustreError: 104823:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2819.795914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2819.803621] LustreError: 104823:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2820.178384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2822.633736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2822.647749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.667371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2825.188559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.190233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2825.212628] Lustre: Skipped 1 previous similar message [ 2825.233547] Lustre: Skipped 2 previous similar messages [ 2828.938428] Lustre: server umount lustre-MDT0000 complete [ 2830.308410] LustreError: 103657:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.339307] LustreError: 103657:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 2832.358547] LustreError: 103656:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.387530] LustreError: 103656:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 2832.409291] LustreError: 103633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748751 with bad export cookie 14911992866008596441 [ 2832.414725] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2832.419467] LustreError: 103633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2832.433572] Lustre: 103647:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2832.450877] LustreError: 104087:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 2832.461253] LustreError: 104087:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 2832.731487] Lustre: server umount lustre-MDT0001 complete [ 2847.014981] Lustre: server umount lustre-OST0000 complete [ 2855.284550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2855.667131] LustreError: 105892:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.734107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2859.938939] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2861.045774] LustreError: 105893:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.195119] LustreError: 105892:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.876236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2868.212195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2872.269138] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2878.832553] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2883.934956] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2889.339622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2889.743409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2894.926551] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2900.013737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2901.383653] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2909.154047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2909.159469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.174431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2914.116511] Lustre: server umount lustre-OST0000 complete [ 2919.396221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2919.402488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.414706] Lustre: Skipped 1 previous similar message [ 2919.418237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2919.422835] Lustre: Skipped 2 previous similar messages [ 2922.896507] Lustre: server umount lustre-MDT0000 complete [ 2925.446076] LustreError: 105872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748844 with bad export cookie 14911992866008597141 [ 2925.450782] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2925.454925] LustreError: 105872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2925.679696] Lustre: server umount lustre-MDT0001 complete [ 2933.573464] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2935.556852] Key type lgssc unregistered [ 2935.807400] LNet: 109423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2935.817449] LNetError: 109423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2935.834205] LNet: Removed LNI 192.168.206.151@tcp [ 2936.510967] Key type .llcrypt unregistered [ 2936.516218] Key type ._llcrypt unregistered [ 2953.791481] Key type ._llcrypt registered [ 2953.793491] Key type .llcrypt registered [ 2953.848285] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2954.433438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2954.554454] alg: No test for adler32 (adler32-zlib) [ 2955.590637] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 2955.795522] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2957.455226] Key type lgssc registered [ 2958.197559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2964.161338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2964.177468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.484728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.583646] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2974.722307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.001291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2977.988581] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2982.835765] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2986.652246] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2991.932814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2992.195632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2996.670514] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3001.939523] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3011.690400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3015.765987] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3018.112532] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3021.283965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3021.292305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.304331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3022.313435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3022.313886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3022.317507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3022.328667] Lustre: Skipped 1 previous similar message [ 3023.042756] Lustre: server umount lustre-OST0000 complete [ 3026.402575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3026.408034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.421129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3032.545333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3032.557131] Lustre: Skipped 3 previous similar messages [ 3037.664370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3037.670647] Lustre: Skipped 1 previous similar message [ 3040.223127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3040.372734] Lustre: server umount lustre-MDT0000 complete [ 3042.789189] LustreError: 110814:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.803732] LustreError: 110814:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 3043.031591] LustreError: 113367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788748962 with bad export cookie 3884699500136666160 [ 3043.037814] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3043.043868] LustreError: 113367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3043.325408] Lustre: server umount lustre-MDT0001 complete [ 3051.119841] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3053.233414] Key type lgssc unregistered [ 3053.454203] LNet: 114468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3053.459731] LNetError: 114468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3053.474633] LNet: Removed LNI 192.168.206.151@tcp [ 3054.098145] Key type .llcrypt unregistered [ 3054.101603] Key type ._llcrypt unregistered [ 3065.601701] Key type ._llcrypt registered [ 3065.603856] Key type .llcrypt registered [ 3065.689521] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3066.467883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3066.484997] alg: No test for adler32 (adler32-zlib) [ 3067.482656] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 3067.651690] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3069.295201] Key type lgssc registered [ 3069.980787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3074.751855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3074.766837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.069960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3078.913448] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3084.258546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.486811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3087.120915] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3091.925368] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3096.024615] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3100.875285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3101.097103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3102.124514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3104.765558] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3109.204680] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3117.538726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3117.548714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.562650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3119.584918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3119.589178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3119.593565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.768079] Lustre: server umount lustre-OST0000 complete [ 3124.706713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3124.708073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3124.717518] Lustre: Skipped 1 previous similar message [ 3124.725953] Lustre: Skipped 2 previous similar messages [ 3129.188443] LustreError: 118443:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000ab0438f4) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3129.263375] Lustre: server umount lustre-MDT0000 complete [ 3129.824269] LustreError: 116394:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3129.835668] LustreError: 116394:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 3131.470419] LustreError: 115645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749050 with bad export cookie 8611152253169990972 [ 3131.476887] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3131.481996] LustreError: 115645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3131.661636] Lustre: server umount lustre-MDT0001 complete [ 3138.364843] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3140.270981] Key type lgssc unregistered [ 3140.485349] LNet: 119194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3140.493466] LNetError: 119194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3140.514682] LNet: Removed LNI 192.168.206.151@tcp [ 3141.120428] Key type .llcrypt unregistered [ 3141.122083] Key type ._llcrypt unregistered [ 3156.923088] Key type ._llcrypt registered [ 3156.924849] Key type .llcrypt registered [ 3156.998827] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3157.601734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3157.679121] alg: No test for adler32 (adler32-zlib) [ 3158.654750] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 3158.843705] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3160.479206] Key type lgssc registered [ 3161.147610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3166.285179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3166.299240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.552073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3170.405617] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3175.813529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3176.033764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3178.440920] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3183.033258] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3187.002586] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3191.595563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3191.886215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3192.949115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3196.516471] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3201.333726] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3206.168462] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3208.569371] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3208.575212] Lustre: Skipped 2 previous similar messages [ 3211.823283] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3211.831292] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3211.842413] Lustre: Skipped 2 previous similar messages [ 3211.844715] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3218.976943] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3218.981297] Lustre: Skipped 1 previous similar message [ 3224.249575] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.51@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3224.254664] Lustre: Skipped 2 previous similar messages [ 3230.224481] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.51@tcp 192.168.0.[2,10]@tcp [ 3233.923837] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.51/24@tcp in log lustre-MDT0000 [ 3233.929896] Lustre: Skipped 2 previous similar messages [ 3238.449676] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.51/24@tcp [ 3238.453132] Lustre: Skipped 1 previous similar message [ 3242.979244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3242.987408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3243.004037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3244.654977] Lustre: server umount lustre-OST0000 complete [ 3247.586149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.586304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3247.599124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.609058] Lustre: Skipped 1 previous similar message [ 3252.710063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.715261] Lustre: Skipped 1 previous similar message [ 3253.165539] Lustre: server umount lustre-MDT0000 complete [ 3255.433602] LustreError: 120566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749174 with bad export cookie 10526316675478078248 [ 3255.439395] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3255.446680] LustreError: 120566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3265.084290] Lustre: server umount lustre-MDT0001 complete [ 3272.229535] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3273.969858] Key type lgssc unregistered [ 3274.180568] LNet: 125477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3274.188772] LNetError: 125477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3274.201930] LNet: Removed LNI 192.168.206.151@tcp [ 3274.749119] Key type .llcrypt unregistered [ 3274.752630] Key type ._llcrypt unregistered [ 3293.562961] Key type ._llcrypt registered [ 3293.564993] Key type .llcrypt registered [ 3293.643638] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3294.315474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3294.329303] alg: No test for adler32 (adler32-zlib) [ 3295.283930] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 3295.464834] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3297.082200] Key type lgssc registered [ 3297.928795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3303.488751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3303.504360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3304.763526] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3304.775481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3304.789831] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.51/24@tcp [ 3307.401130] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3312.586778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.774955] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3312.791979] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.51/24@tcp [ 3312.829855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3315.914623] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3320.956304] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3324.759135] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3329.216814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3329.461612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3333.225255] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3337.408641] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3349.926806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3360.227195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3360.230776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3360.231820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3360.237016] Lustre: Skipped 1 previous similar message [ 3365.384470] Lustre: server umount lustre-OST0000 complete [ 3369.441284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3369.446532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.453750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3369.456637] Lustre: Skipped 1 previous similar message [ 3370.464777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3370.464951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3373.613428] Lustre: server umount lustre-MDT0000 complete [ 3375.485794] LustreError: 126992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749294 with bad export cookie 3315846419634697558 [ 3375.489563] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3375.493466] LustreError: 126992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3375.660616] Lustre: server umount lustre-MDT0001 complete [ 3380.874300] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3382.266072] Key type lgssc unregistered [ 3382.443500] LNet: 130616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3382.447077] LNetError: 130616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3382.461234] LNet: Removed LNI 192.168.206.151@tcp [ 3382.892193] Key type .llcrypt unregistered [ 3382.894212] Key type ._llcrypt unregistered [ 3395.360800] Key type ._llcrypt registered [ 3395.362743] Key type .llcrypt registered [ 3395.418640] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 3402.990865] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3403.381921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3403.431439] alg: No test for adler32 (adler32-zlib) [ 3404.323666] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 3404.429578] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3406.039194] Key type lgssc registered [ 3406.526502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3423.087075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3423.096533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3424.242877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3424.265212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3424.314125] Lustre: lustre-MDT0000: new disk, initializing [ 3424.361289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3424.372889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3426.589457] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3432.588656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.643056] Lustre: 133987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3432.673911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3432.677037] Lustre: Skipped 1 previous similar message [ 3432.723310] Lustre: lustre-MDT0001: new disk, initializing [ 3432.777772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3432.801077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3432.805600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3434.723080] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3439.469383] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3441.782256] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3445.033030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3445.188050] Lustre: lustre-OST0000: new disk, initializing [ 3445.191819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3445.196658] Lustre: 135954:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3445.245151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3448.352610] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3453.551733] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3454.486169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3454.491405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3454.535673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3455.609233] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3455.703752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3457.483375] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3457.576423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3459.553498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3459.555066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.557148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3459.561276] Lustre: Skipped 1 previous similar message [ 3460.513970] Lustre: server umount lustre-OST0000 complete [ 3463.648586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3463.651976] LustreError: Skipped 1 previous similar message [ 3463.653976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.661242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3463.663881] Lustre: Skipped 1 previous similar message [ 3464.674619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3464.677087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.338991] Lustre: server umount lustre-MDT0000 complete [ 3470.203894] LustreError: 135953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749389 with bad export cookie 8606807336110849287 [ 3470.206097] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3470.210721] LustreError: 135953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3470.404961] Lustre: server umount lustre-MDT0001 complete [ 3474.742218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.963280] LustreError: 138167:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3474.973397] LustreError: 138167:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 3474.998739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3476.719221] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3480.032309] LustreError: 138168:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3480.541370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.717229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3482.453380] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3485.935272] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3488.495390] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3491.625422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3491.777223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3494.519684] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3497.931940] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3499.800721] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3500.921451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3502.748231] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3505.941319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 3510.188151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3510.264000] Lustre: lustre-OST0001: new disk, initializing [ 3510.266396] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3510.269937] Lustre: 141566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3510.303528] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3512.886424] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3516.947535] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3518.487221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3518.491997] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3518.527301] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3519.131445] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3520.978468] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3521.070482] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3523.442941] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89add0aa7800.ost_server_uuid 50 [ 3524.175348] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89add0aa7800.ost_server_uuid in FULL state after 0 sec [ 3528.672455] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3528.672774] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3528.676405] LustreError: Skipped 1 previous similar message [ 3528.683206] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3528.686604] Lustre: Skipped 2 previous similar messages [ 3533.458760] Lustre: server umount lustre-OST0001 complete [ 3533.797505] LustreError: 141180:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.805073] LustreError: 141180:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3535.840350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3535.840648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.844774] LustreError: Skipped 1 previous similar message [ 3535.850534] Lustre: Skipped 1 previous similar message [ 3535.854244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3535.861080] Lustre: Skipped 1 previous similar message [ 3538.913699] LustreError: 139989:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.921060] LustreError: 139989:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 3541.473487] Lustre: server umount lustre-OST0000 complete [ 3544.033986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3544.034669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3544.041415] Lustre: Skipped 1 previous similar message [ 3544.044573] Lustre: Skipped 2 previous similar messages [ 3549.365778] Lustre: server umount lustre-MDT0000 complete [ 3551.200768] LustreError: 141564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749470 with bad export cookie 8606807336110850169 [ 3551.205050] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3551.205866] LustreError: 141564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3551.397520] Lustre: server umount lustre-MDT0001 complete [ 3560.246972] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3561.744724] Key type lgssc unregistered [ 3561.905606] LNet: 144578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3561.909953] LNetError: 144578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3561.922081] LNet: Removed LNI 192.168.206.151@tcp [ 3562.350904] Key type .llcrypt unregistered [ 3562.352913] Key type ._llcrypt unregistered [ 3580.968508] Key type ._llcrypt registered [ 3580.969715] Key type .llcrypt registered [ 3581.025689] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 3588.837557] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3589.423169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3589.431592] alg: No test for adler32 (adler32-zlib) [ 3590.337538] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 3590.472677] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3592.103201] Key type lgssc registered [ 3592.672438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3609.725083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3609.733810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3610.838383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3610.853158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3610.895419] Lustre: lustre-MDT0000: new disk, initializing [ 3610.933111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3610.943803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3612.674245] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3618.115174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.162800] Lustre: 148699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3618.193492] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3618.196043] Lustre: Skipped 1 previous similar message [ 3618.237549] Lustre: lustre-MDT0001: new disk, initializing [ 3618.278620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3618.295691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3618.300559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3620.269139] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3625.527645] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3628.339725] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3631.702167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3631.830183] Lustre: lustre-OST0000: new disk, initializing [ 3631.833347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3631.838435] Lustre: 150666:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3631.881123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.457480] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3637.766379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3637.770847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3637.822250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3639.252427] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3641.610238] Lustre: Failing over lustre-OST0000 [ 3641.691194] Lustre: server umount lustre-OST0000 complete [ 3656.019694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3656.235643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3656.246427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3656.258239] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3658.335190] Lustre: 146926:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788749561/real 1788749561] req@00000000c50c61bc x1875639805176576/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788749577 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3659.801677] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3661.031262] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3661.279496] Lustre: 146929:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788749564/real 1788749564] req@0000000087e8723d x1875639805176960/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788749580 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3661.296993] Lustre: 146929:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 3661.325030] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3661.330536] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3662.131894] Lustre: Failing over lustre-MDT0000 [ 3662.259595] Lustre: server umount lustre-MDT0000 complete [ 3664.160129] Lustre: 146928:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788749567/real 1788749567] req@00000000665fe826 x1875639805177856/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788749583 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3664.356734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3664.362111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3664.371574] Lustre: Skipped 1 previous similar message [ 3666.402980] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.423354] Lustre: Skipped 1 previous similar message [ 3669.535143] Lustre: 146929:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788749572/real 1788749572] req@00000000f9ab23bc x1875639805179008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788749588 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3671.243291] LustreError: 149503:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3671.253585] LustreError: 149503:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 6 previous similar messages [ 3676.364359] LustreError: 149503:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.372836] LustreError: 149503:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 3677.551697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.634694] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3677.640953] Lustre: 148702:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3677.648258] LustreError: 148711:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3677.667906] LustreError: 148711:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 5 previous similar messages [ 3677.805592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.850678] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3680.398713] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3681.484418] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3683.304694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3683.306795] Lustre: Skipped 1 previous similar message [ 3683.317080] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3683.338448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3686.881100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3686.887787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3686.900276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3688.417801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3688.425038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3688.432810] Lustre: Skipped 1 previous similar message [ 3691.530601] Lustre: server umount lustre-OST0000 complete [ 3698.656842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3698.656913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3698.657966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3698.670722] Lustre: Skipped 2 previous similar messages [ 3699.805882] Lustre: server umount lustre-MDT0000 complete [ 3701.545633] LustreError: 149494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749620 with bad export cookie 7767842103984404333 [ 3701.548422] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3701.552802] LustreError: 149494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3701.707972] Lustre: server umount lustre-MDT0001 complete [ 3706.541738] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3707.927873] Key type lgssc unregistered [ 3708.090904] LNet: 154310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3708.095964] LNetError: 154310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3708.106946] LNet: Removed LNI 192.168.206.151@tcp [ 3708.520388] Key type .llcrypt unregistered [ 3708.522292] Key type ._llcrypt unregistered [ 3731.924459] Key type ._llcrypt registered [ 3731.925954] Key type .llcrypt registered [ 3731.973951] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3732.458146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3732.482316] alg: No test for adler32 (adler32-zlib) [ 3733.368872] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 3733.470497] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3735.071234] Key type lgssc registered [ 3735.573196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3738.997105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3739.010204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3740.279473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3742.283908] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3747.187682] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3749.595848] LustreError: 155853:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3749.600625] LustreError: 155853:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3749.604377] LustreError: 155853:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 3749.765469] Lustre: server umount lustre-MDT0000 complete [ 3757.091207] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 3761.274844] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3776.189668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3776.290153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3776.303950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3776.343781] Lustre: lustre-MDT0000: new disk, initializing [ 3776.383926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3776.393922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3778.080868] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3783.258067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.306881] Lustre: 159165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3783.339257] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3783.341700] Lustre: Skipped 1 previous similar message [ 3783.381736] Lustre: lustre-MDT0001: new disk, initializing [ 3783.422888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3783.438231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3783.443512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3785.284471] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3790.142407] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3792.638506] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3795.777263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3795.907850] Lustre: lustre-OST0000: new disk, initializing [ 3795.910856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3795.915225] Lustre: 161132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3795.954928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3798.462667] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3803.214136] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3806.219547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3806.228037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3806.271117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3811.633986] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3812.727881] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:55:32 (1788749732) [ 3942.368375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3942.373699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.380899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3944.416329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3944.416442] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3944.421619] Lustre: Skipped 1 previous similar message [ 3944.435782] Lustre: Skipped 1 previous similar message [ 3945.907558] Lustre: server umount lustre-MDT0000 complete [ 3948.430900] LustreError: 159156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749867 with bad export cookie 11078416680418584341 [ 3948.433232] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3948.438276] LustreError: 159156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3949.535704] LustreError: 159171:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.536081] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.544315] LustreError: 159171:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 3949.554463] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3953.568245] LustreError: 162900:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.569054] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3954.651828] Lustre: server umount lustre-MDT0001 complete [ 3962.964817] Lustre: server umount lustre-OST0000 complete [ 3966.909472] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 3970.952557] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3984.501309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3984.595142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3984.608689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3984.648916] Lustre: lustre-MDT0000: new disk, initializing [ 3984.683223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3984.695122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3986.220703] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3990.651663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3990.692664] Lustre: 165767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3990.713866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3990.715804] Lustre: Skipped 1 previous similar message [ 3990.750919] Lustre: lustre-MDT0001: new disk, initializing [ 3990.800427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3990.804777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3992.328989] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3996.701941] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3999.067632] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4001.985310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4002.105511] Lustre: lustre-OST0000: new disk, initializing [ 4002.107863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4002.112988] Lustre: 167737:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4002.160424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.163409] Lustre: Skipped 1 previous similar message [ 4004.793661] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4009.471243] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4009.985904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4009.992236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4010.046273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4011.274508] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4011.366257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4013.096749] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4013.191798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4015.074693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4015.075154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.078414] LustreError: Skipped 1 previous similar message [ 4015.079424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4015.084518] Lustre: Skipped 1 previous similar message [ 4020.126395] Lustre: server umount lustre-OST0000 complete [ 4021.732344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4021.737924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4025.312684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4025.315843] Lustre: Skipped 3 previous similar messages [ 4036.063106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4036.133495] Lustre: server umount lustre-MDT0000 complete [ 4037.692588] LustreError: 165759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788749956 with bad export cookie 11078416680418743157 [ 4037.696083] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4037.697734] LustreError: 165759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4037.840335] Lustre: server umount lustre-MDT0001 complete [ 4041.286970] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:59:21 (1788749961) [ 4047.325988] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 4051.083888] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4065.929819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.040788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4066.059121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4066.100662] Lustre: lustre-MDT0000: new disk, initializing [ 4066.149910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4066.159236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4067.940297] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4073.059511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.116455] Lustre: 172493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4073.241909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4075.001079] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4079.328354] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4081.514309] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4084.294812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4084.422123] Lustre: 174467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4086.793634] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4091.136454] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4094.473697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4094.519472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4099.552478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4099.554557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.557647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4099.562086] Lustre: Skipped 2 previous similar messages [ 4099.564925] Lustre: Skipped 4 previous similar messages [ 4100.858810] Lustre: server umount lustre-OST0000 complete [ 4104.161535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4104.166771] LustreError: Skipped 1 previous similar message [ 4104.674135] LustreError: 172499:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4106.713699] Lustre: server umount lustre-MDT0000 complete [ 4109.479723] LustreError: 173292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750028 with bad export cookie 11078416680418744018 [ 4109.487749] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4109.488652] LustreError: 173292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4109.767591] Lustre: server umount lustre-MDT0001 complete [ 4117.709803] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:00:36 (1788750036) [ 4136.945921] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 4149.903306] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4194.392595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4194.842148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4194.870805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4194.881328] Lustre: Skipped 3 previous similar messages [ 4195.012261] Lustre: lustre-MDT0000: new disk, initializing [ 4195.015396] Lustre: Skipped 2 previous similar messages [ 4195.117624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4195.123169] Lustre: Skipped 2 previous similar messages [ 4195.161426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4195.169551] Lustre: Skipped 2 previous similar messages [ 4201.952499] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4212.493111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4212.576420] Lustre: 178802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4212.823362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4215.940945] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4224.043220] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4227.939266] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4232.891557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4233.086028] Lustre: 180782:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4237.442112] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4242.601884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4244.524871] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4252.640460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4252.651265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.663156] Lustre: Skipped 2 previous similar messages [ 4252.666825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4252.674926] Lustre: Skipped 2 previous similar messages [ 4255.468840] Lustre: server umount lustre-OST0000 complete [ 4264.448340] Lustre: server umount lustre-MDT0000 complete [ 4267.361721] LustreError: 178796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750186 with bad export cookie 11078416680418745089 [ 4267.371205] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4267.722923] Lustre: server umount lustre-MDT0001 complete [ 4277.234963] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 4279.553969] Key type lgssc unregistered [ 4279.788732] LNet: 182633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4279.794484] LNetError: 182633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4279.815565] LNet: Removed LNI 192.168.206.151@tcp [ 4280.513538] Key type .llcrypt unregistered [ 4280.516331] Key type ._llcrypt unregistered [ 4299.543772] Key type ._llcrypt registered [ 4299.545909] Key type .llcrypt registered [ 4299.633980] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4300.393735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4300.424334] alg: No test for adler32 (adler32-zlib) [ 4301.444433] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 4301.617250] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 4303.314739] Key type lgssc registered [ 4304.161545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4310.285247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4310.312194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.623379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4314.856367] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4321.224375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4321.581801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4324.940351] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4330.801815] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4335.372533] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4341.238491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4341.482666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4346.841848] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4354.122282] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4357.877657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4377.568854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4377.576253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4377.594521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4378.595392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4378.601077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4378.605770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4381.684521] Lustre: server umount lustre-OST0000 complete [ 4388.320246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4388.332849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.350432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4388.352992] Lustre: Skipped 1 previous similar message [ 4391.463534] Lustre: server umount lustre-MDT0000 complete [ 4393.959627] LustreError: 184025:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.979462] LustreError: 184025:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 4394.803859] LustreError: 184005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750313 with bad export cookie 1225820427978401567 [ 4394.814225] LustreError: 184005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4394.815441] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4395.278973] Lustre: server umount lustre-MDT0001 complete [ 4404.460755] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 4406.976910] Key type lgssc unregistered [ 4407.277304] LNet: 187575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4407.291792] LNetError: 187575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4407.314172] LNet: Removed LNI 192.168.206.151@tcp [ 4408.138281] Key type .llcrypt unregistered [ 4408.140848] Key type ._llcrypt unregistered [ 4429.679545] Key type ._llcrypt registered [ 4429.683631] Key type .llcrypt registered [ 4429.792541] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4430.700191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4430.736213] alg: No test for adler32 (adler32-zlib) [ 4431.730826] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 4431.930247] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 4433.583186] Key type lgssc registered [ 4434.689493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4442.978530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4443.005041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4444.365737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4448.177425] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4454.883522] LustreError: 188990:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.909225] LustreError: 188990:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 4456.234901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.548825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4461.259745] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4469.578657] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4474.870876] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4481.959355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4482.209280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4488.599609] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4489.404926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4496.359778] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4507.116406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.120449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4507.145825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4507.157851] Lustre: Skipped 1 previous similar message [ 4508.712314] Lustre: server umount lustre-OST0000 complete [ 4515.874879] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff89add1d16000.ost_server_uuid 50 [ 4529.263719] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4543.457199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4543.468046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.487039] Lustre: Skipped 1 previous similar message [ 4543.495089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4547.625044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4547.629100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4549.434697] Lustre: server umount lustre-MDT0000 complete [ 4552.673812] LustreError: 188986:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4552.694072] LustreError: 188986:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4552.987543] LustreError: 188971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750472 with bad export cookie 10864085306570525931 [ 4552.996031] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4552.999104] LustreError: 188971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4553.422590] Lustre: server umount lustre-MDT0001 complete [ 4562.493681] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:08:01 (1788750481) [ 4569.828997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4570.421080] LustreError: 192926:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4570.535229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4575.138603] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4575.723195] LustreError: 192927:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4580.836522] LustreError: 192926:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4583.679297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.023219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4587.929705] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4596.822907] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4604.875913] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4613.005397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4613.484914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4615.546144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4620.585372] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4629.121715] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4636.472769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4636.737748] Lustre: lustre-OST0001: new disk, initializing [ 4636.741499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4636.747852] Lustre: 195768:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4636.811670] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4643.462603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4643.476099] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4643.584790] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4644.067033] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4653.492448] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4656.609533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4656.614430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4656.632586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4656.646585] Lustre: Skipped 1 previous similar message [ 4662.401308] Lustre: server umount lustre-OST0000 complete [ 4664.294849] LustreError: 194749:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.311982] LustreError: 194749:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 4667.982949] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4668.252891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4672.777882] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4672.952357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4684.782495] LustreError: 196079:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4684.803192] LustreError: 196079:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 11 previous similar messages [ 4689.891204] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4689.903884] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.909682] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4689.927808] Lustre: Skipped 2 previous similar messages [ 4689.934508] Lustre: Skipped 3 previous similar messages [ 4692.423924] Lustre: server umount lustre-OST0001 complete [ 4700.130941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4700.131769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4700.148043] Lustre: Skipped 1 previous similar message [ 4700.153174] Lustre: Skipped 1 previous similar message [ 4703.177349] Lustre: server umount lustre-MDT0000 complete [ 4706.638766] LustreError: 195767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750625 with bad export cookie 10864085306570527002 [ 4706.645365] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4706.647858] LustreError: 195767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4706.970316] Lustre: server umount lustre-MDT0001 complete [ 4734.088290] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:10:53 (1788750653) [ 4743.389290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4743.730406] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4743.795454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4744.043613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4749.980993] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4759.768757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4759.841898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4759.863222] Lustre: 199206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4764.554323] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4773.123279] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4779.361865] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4787.015661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4787.323748] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4795.136603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4795.240904] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4804.109283] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4811.550088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4811.646577] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4811.942024] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4811.949177] Lustre: Skipped 2 previous similar messages [ 4819.313182] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4827.090582] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4831.203965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4831.205609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4831.211047] LustreError: Skipped 2 previous similar messages [ 4831.216368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4831.223725] Lustre: Skipped 1 previous similar message [ 4836.022688] Lustre: server umount lustre-OST0000 complete [ 4838.098515] LustreError: 201053:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4838.116063] LustreError: 201053:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 4853.728220] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4858.657968] Lustre: server umount lustre-OST0001 complete [ 4863.968961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4863.975400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4863.990791] Lustre: Skipped 3 previous similar messages [ 4863.999513] Lustre: Skipped 7 previous similar messages [ 4869.141455] Lustre: server umount lustre-MDT0000 complete [ 4872.673103] LustreError: 199199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750791 with bad export cookie 10864085306570528409 [ 4872.674778] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4872.688037] LustreError: 199199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4873.012618] Lustre: server umount lustre-MDT0001 complete [ 4895.716555] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:13:34 (1788750814) [ 4912.879379] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 4923.887499] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4959.733859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4959.954417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4959.978340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4960.058349] Lustre: lustre-MDT0000: new disk, initializing [ 4960.133445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4960.152408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4964.363719] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4974.892843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4975.036620] Lustre: 207302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4975.081382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4975.085079] Lustre: Skipped 1 previous similar message [ 4975.152755] Lustre: lustre-MDT0001: new disk, initializing [ 4975.329932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4975.340342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4980.009357] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4989.710109] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4995.288335] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5002.745088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5003.038318] Lustre: lustre-OST0000: new disk, initializing [ 5003.041615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5003.051285] Lustre: 209268:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5009.451851] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5009.466413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5009.476906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5009.571309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5019.106517] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5023.918630] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5024.199090] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5028.909800] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5029.085573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5034.981761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5034.993518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5034.997701] Lustre: Skipped 1 previous similar message [ 5035.012868] Lustre: Skipped 1 previous similar message [ 5045.217143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5045.427581] Lustre: server umount lustre-OST0000 complete [ 5063.136311] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5063.427993] Lustre: server umount lustre-MDT0000 complete [ 5065.697464] LustreError: 207313:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5065.734953] LustreError: 207313:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 13 previous similar messages [ 5066.861963] LustreError: 207292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788750985 with bad export cookie 10864085306570529970 [ 5066.876141] LustreError: 207292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5066.881023] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5067.242131] Lustre: server umount lustre-MDT0001 complete [ 5075.962693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5080.970239] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5088.884207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5093.573180] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5100.390520] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5105.359976] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5111.112921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5117.017307] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5122.911644] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5126.374470] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5126.536805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5129.810389] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5129.988272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5132.266899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5132.274456] LustreError: Skipped 1 previous similar message [ 5137.596155] Lustre: server umount lustre-OST0000 complete [ 5142.230724] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5142.387958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5146.165667] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5174.264632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5180.473180] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5187.832035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.862750] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.864654] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5188.876542] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5193.832667] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 5198.304831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5198.308277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5198.318078] LustreError: Skipped 1 previous similar message [ 5198.326040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5198.326049] Lustre: Skipped 14 previous similar messages [ 5198.366972] Lustre: Skipped 5 previous similar messages [ 5204.125470] Lustre: server umount lustre-OST0000 complete [ 5213.647484] Lustre: server umount lustre-MDT0000 complete [ 5213.682145] LustreError: 211490:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5213.707995] LustreError: 211490:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 5217.694811] LustreError: 211469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751136 with bad export cookie 10864085306570530852 [ 5217.706384] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5217.719118] LustreError: 211469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5218.231430] Lustre: server umount lustre-MDT0001 complete [ 5225.885790] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:19:05 (1788751145) [ 5230.991163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5231.388432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5231.393981] Lustre: Skipped 6 previous similar messages [ 5235.133889] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5242.902927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5247.228181] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5253.807695] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5258.206472] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5263.234390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5268.016535] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5273.263224] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5276.739711] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5277.961044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5281.184251] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5281.355576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5286.492321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5286.627426] Lustre: lustre-OST0001: new disk, initializing [ 5286.632222] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5286.636784] Lustre: 220845:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5290.946464] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5296.204930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5296.212920] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5296.285200] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5298.297495] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5301.824200] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5302.027194] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5305.490476] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5305.643191] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5311.978871] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5311.986383] LustreError: Skipped 1 previous similar message [ 5321.183329] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5321.312348] Lustre: server umount lustre-OST0001 complete [ 5327.211589] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5327.412490] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5331.710893] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5331.906664] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5359.445720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5364.483179] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5364.715106] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5364.733271] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5364.733864] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5364.753064] Lustre: Skipped 1 previous similar message [ 5370.736578] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5372.169974] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5388.255992] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5388.336161] Lustre: server umount lustre-OST0001 complete [ 5410.190720] Lustre: server umount lustre-OST0000 complete [ 5419.657972] Lustre: server umount lustre-MDT0000 complete [ 5422.510146] LustreError: 217451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751341 with bad export cookie 10864085306570531811 [ 5422.517225] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5422.525367] LustreError: 217451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5428.830743] Lustre: server umount lustre-MDT0001 complete [ 5445.228869] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:22:44 (1788751364) [ 5450.183070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5450.358866] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5450.384540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5453.931338] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5461.195217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5461.276405] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5461.309686] Lustre: 226084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5464.886270] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5470.867396] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5475.141604] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5479.837858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5479.966225] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5484.465241] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5489.446952] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5495.993856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5496.069636] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5500.893451] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5505.954592] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5506.976439] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5510.124756] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5510.291647] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5513.118708] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5513.289269] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5516.948793] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89adc5d27000.ost_server_uuid 50 [ 5517.935607] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89adc5d27000.ost_server_uuid in FULL state after 0 sec [ 5518.597911] Lustre: Permanently deactivating lustre-OST0001 [ 5518.612703] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5520.920711] Lustre: Permanently reactivating lustre-OST0001 [ 5528.035586] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5528.035772] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5528.050972] LustreError: Skipped 4 previous similar messages [ 5528.051754] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5528.051759] Lustre: Skipped 21 previous similar messages [ 5528.092756] Lustre: Skipped 11 previous similar messages [ 5529.553502] Lustre: server umount lustre-OST0001 complete [ 5533.161042] LustreError: 227931:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5533.182847] LustreError: 227931:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 38 previous similar messages [ 5538.093735] Lustre: server umount lustre-OST0000 complete [ 5555.167174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5555.362837] Lustre: server umount lustre-MDT0000 complete [ 5558.039361] LustreError: 228958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751477 with bad export cookie 10864085306570533106 [ 5558.043183] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5558.055193] LustreError: 228958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5558.314082] Lustre: server umount lustre-MDT0001 complete [ 5576.798244] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:24:56 (1788751496) [ 5582.835084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5583.054728] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5583.084539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5583.096381] Lustre: Skipped 5 previous similar messages [ 5587.096316] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5592.651511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.715931] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5592.734328] Lustre: 232774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5595.490172] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5600.417201] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5604.768562] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5610.324444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5610.536630] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5610.558522] Lustre: Permanently deactivating lustre-OST0000 [ 5610.581928] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5615.591298] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5616.151599] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5621.438473] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5627.040399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5627.132759] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5632.376972] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5633.199662] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5639.025203] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5640.448717] Lustre: Permanently reactivating lustre-OST0000 [ 5640.466141] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5640.469672] Lustre: Skipped 2 previous similar messages [ 5649.840277] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 33 seconds. I think it's dead, and I am evicting it. exp 000000001417047d, cur 1788751568 deadline 1788751565 last 1788751535 [ 5649.983206] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5649.988686] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.998418] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5650.002441] Lustre: Skipped 1 previous similar message [ 5650.050805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5657.707557] Lustre: server umount lustre-OST0001 complete [ 5666.898469] Lustre: server umount lustre-OST0000 complete [ 5684.191351] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5684.333642] Lustre: server umount lustre-MDT0000 complete [ 5687.270317] LustreError: 232768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751606 with bad export cookie 10864085306570534751 [ 5687.281378] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5687.643625] Lustre: server umount lustre-MDT0001 complete [ 5697.423442] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 5700.093455] Key type lgssc unregistered [ 5700.418869] LNet: 237630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5700.425193] LNetError: 237630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5701.482476] LNet: Removed LNI 192.168.206.151@tcp [ 5702.607409] Key type .llcrypt unregistered [ 5702.610128] Key type ._llcrypt unregistered [ 5720.881743] Key type ._llcrypt registered [ 5720.883539] Key type .llcrypt registered [ 5720.998320] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 5721.736448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5721.820319] alg: No test for adler32 (adler32-zlib) [ 5722.844252] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 5723.036034] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 5724.687206] Key type lgssc registered [ 5725.635045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5733.300715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5733.321164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.641425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5738.213601] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5744.106379] LustreError: 239074:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5744.123497] LustreError: 239074:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 5745.510728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5745.623662] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5745.661568] Lustre: 239062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5745.669650] Lustre: Permanently deactivating lustre-MDT0001 [ 5745.688264] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5745.942604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5749.472641] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5749.476941] LustreError: 239104:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5749.488857] LustreError: 239104:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5749.494350] LustreError: 239104:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 5750.052592] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5756.293851] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5760.790337] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5766.520505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5766.778026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.853185] LustreError: 240914:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.445440] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5776.359203] LustreError: 241217:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.356659] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5779.428353] LustreError: 240913:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5779.447672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5779.448990] LustreError: 240913:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 5782.768979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5782.984481] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5787.743851] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5788.152576] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5788.161091] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5793.553464] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5796.010736] Lustre: Permanently reactivating lustre-MDT0001 [ 5796.022447] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5796.026126] Lustre: Skipped 2 previous similar messages [ 5803.748976] Lustre: lustre-MDT0001: Client 205a663b-5f78-4d51-a753-102fe9504a94 (at 192.168.206.51@tcp) reconnecting [ 5809.530181] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5810.824247] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5815.383119] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5815.574865] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5819.596418] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5819.814166] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5822.208986] Lustre: Permanently deactivating lustre-MDT0001 [ 5822.234885] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5822.239449] Lustre: Skipped 2 previous similar messages [ 5832.299669] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5841.379159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5841.380906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5841.387707] Lustre: Skipped 2 previous similar messages [ 5841.393658] Lustre: Skipped 3 previous similar messages [ 5845.403879] Lustre: server umount lustre-MDT0000 complete [ 5846.498663] LustreError: 243626:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5846.517526] LustreError: 243626:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 5848.442420] LustreError: 239053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751767 with bad export cookie 13450067613177206507 [ 5848.444238] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5848.456832] LustreError: 239053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5848.764061] Lustre: server umount lustre-MDT0001 complete [ 5854.496825] Lustre: server umount lustre-OST0000 complete [ 5864.260870] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:29:43 (1788751783) [ 5872.779989] Lustre: server umount lustre-OST0001 complete [ 5878.715313] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 5886.311985] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 5919.783557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.975993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5919.983447] Lustre: Skipped 2 previous similar messages [ 5920.003387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5920.082494] Lustre: lustre-MDT0000: new disk, initializing [ 5920.142966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5920.157148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5924.293921] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5934.402865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5934.506494] Lustre: 247378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5934.566821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5934.574387] Lustre: Skipped 1 previous similar message [ 5934.633203] Lustre: lustre-MDT0001: new disk, initializing [ 5934.706329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5934.747405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5934.754586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5938.679801] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5946.975368] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5951.446169] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5957.416619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5957.692205] Lustre: lustre-OST0000: new disk, initializing [ 5957.695636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5957.703989] Lustre: 249349:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5957.775349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5962.987730] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5963.853745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5963.863811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5963.981233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5971.350935] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5973.388492] LustreError: 248512:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5980.934849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5981.082667] Lustre: lustre-OST0001: new disk, initializing [ 5981.085639] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5981.097065] Lustre: 250461:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5983.463344] LustreError: 248512:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5983.494798] LustreError: 247384:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5986.560684] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5987.416385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5987.430059] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5987.499536] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5993.495226] LustreError: 247384:0:(mdt_handler.c:3467:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5995.881559] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5997.525595] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6002.411756] Lustre: lustre-OST0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 6003.174769] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6003.178522] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.189527] LustreError: Skipped 1 previous similar message [ 6003.222838] Lustre: Skipped 2 previous similar messages [ 6005.611117] Lustre: server umount lustre-OST0001 complete [ 6007.500403] LustreError: 249765:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6023.654298] LustreError: 249765:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6023.661629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.666378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6023.666388] Lustre: Skipped 2 previous similar messages [ 6023.672695] LustreError: 249765:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [ 6023.700672] Lustre: Skipped 1 previous similar message [ 6028.153833] Lustre: server umount lustre-OST0000 complete [ 6031.840526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6031.849625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6031.869692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6031.875532] Lustre: Skipped 1 previous similar message [ 6037.526981] Lustre: server umount lustre-MDT0000 complete [ 6041.446779] LustreError: 247368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788751960 with bad export cookie 13450067613177209111 [ 6041.453338] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6041.460647] LustreError: 247368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6047.814040] Lustre: server umount lustre-MDT0001 complete [ 6057.521979] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6059.998788] Key type lgssc unregistered [ 6060.271647] LNet: 252494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6060.281944] LNetError: 252494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6060.291601] LNet: Removed LNI 192.168.206.151@tcp [ 6061.064248] Key type .llcrypt unregistered [ 6061.068130] Key type ._llcrypt unregistered [ 6098.319127] Key type ._llcrypt registered [ 6098.320980] Key type .llcrypt registered [ 6098.429995] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6099.329193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6099.365204] alg: No test for adler32 (adler32-zlib) [ 6100.409400] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 6100.606256] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6102.263262] Key type lgssc registered [ 6103.161974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6109.874958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6109.888612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.113408] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6111.137988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6111.267549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6115.116154] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6123.798189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6123.907352] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6123.956743] Lustre: 254699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6124.258540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6128.278600] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6134.998276] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6140.265081] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6146.416114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6146.663688] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6146.853501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6152.724175] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6156.467594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6158.808332] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6207.459447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6207.465446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6207.479640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6207.982472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6207.989479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6208.004268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6212.263984] Lustre: server umount lustre-OST0000 complete [ 6215.668341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6226.274278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6226.482346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6228.518034] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.526477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.527557] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6228.542672] Lustre: Skipped 1 previous similar message [ 6231.309493] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6237.412522] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6279.648155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6279.652563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6279.659755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6279.664177] Lustre: Skipped 1 previous similar message [ 6279.676311] Lustre: Skipped 1 previous similar message [ 6280.805298] Lustre: server umount lustre-OST0000 complete [ 6284.769603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6284.771757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6284.779501] Lustre: Skipped 1 previous similar message [ 6284.792062] Lustre: Skipped 1 previous similar message [ 6289.877718] Lustre: server umount lustre-MDT0000 complete [ 6292.640692] LustreError: 254689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752211 with bad export cookie 12477641259973649796 [ 6292.647267] LustreError: 254689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6292.647783] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6292.895584] Lustre: server umount lustre-MDT0001 complete [ 6299.464949] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6301.499358] Key type lgssc unregistered [ 6301.727526] LNet: 260115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6301.735736] LNetError: 260115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6301.756306] LNet: Removed LNI 192.168.206.151@tcp [ 6302.385544] Key type .llcrypt unregistered [ 6302.387893] Key type ._llcrypt unregistered [ 6318.193818] Key type ._llcrypt registered [ 6318.196215] Key type .llcrypt registered [ 6318.290434] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6318.975367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6319.012109] alg: No test for adler32 (adler32-zlib) [ 6319.991366] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 6320.169878] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6321.807224] Key type lgssc registered [ 6322.691872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6328.297782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6328.328191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.652924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6332.508924] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6338.231288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6338.527560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6341.684445] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6346.765311] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6351.068955] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6355.952262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6356.218854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6360.248788] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6365.384830] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6374.643467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6393.322342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.348597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6397.303580] Lustre: server umount lustre-OST0000 complete [ 6403.552706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.562321] Lustre: Skipped 1 previous similar message [ 6403.567849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6403.571310] Lustre: Skipped 2 previous similar messages [ 6408.331891] Lustre: server umount lustre-MDT0000 complete [ 6408.673762] LustreError: 261506:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6408.691943] LustreError: 261506:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 6410.723938] LustreError: 261486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752329 with bad export cookie 15441732265642640705 [ 6410.728357] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6410.732356] LustreError: 261486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6410.951268] Lustre: server umount lustre-MDT0001 complete [ 6418.932580] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6421.146456] Key type lgssc unregistered [ 6421.381069] LNet: 265827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6421.392175] LNetError: 265827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6421.410187] LNet: Removed LNI 192.168.206.151@tcp [ 6422.040193] Key type .llcrypt unregistered [ 6422.042242] Key type ._llcrypt unregistered [ 6434.606777] Key type ._llcrypt registered [ 6434.609705] Key type .llcrypt registered [ 6434.688658] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6435.592679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6435.640305] alg: No test for adler32 (adler32-zlib) [ 6436.633198] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 6436.864184] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6438.552122] Key type lgssc registered [ 6439.441223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6445.623890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6445.643744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6446.935347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.995390] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6456.946080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6457.270678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6460.592479] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6465.896521] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6469.859982] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6475.079777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6475.328594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6479.843253] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6484.787363] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6493.741676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6502.881715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6502.889171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6502.899473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6503.908384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6503.908723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6503.912332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6503.927550] Lustre: Skipped 1 previous similar message [ 6507.728964] Lustre: server umount lustre-OST0000 complete [ 6513.633893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6513.638620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6513.647958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6516.638855] Lustre: server umount lustre-MDT0000 complete [ 6519.007716] LustreError: 267006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752438 with bad export cookie 15401305422522252434 [ 6519.010762] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6519.016568] LustreError: 267006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6519.217586] Lustre: server umount lustre-MDT0001 complete [ 6524.510770] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6526.020951] Key type lgssc unregistered [ 6526.201624] LNet: 270720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6526.209922] LNetError: 270720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6526.218644] LNet: Removed LNI 192.168.206.151@tcp [ 6526.677303] Key type .llcrypt unregistered [ 6526.679054] Key type ._llcrypt unregistered [ 6539.715332] Key type ._llcrypt registered [ 6539.717114] Key type .llcrypt registered [ 6539.771253] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6540.321061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6540.403651] alg: No test for adler32 (adler32-zlib) [ 6541.318303] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 6541.481168] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6543.113563] Key type lgssc registered [ 6543.755230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6548.482334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6548.497486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6549.867894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6552.223631] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6557.006232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6557.227815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6559.544390] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6563.277614] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6566.291812] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6570.169054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6570.425228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6573.854540] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6574.505499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6578.141203] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6598.113466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6598.116315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6598.132567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6598.136458] Lustre: Skipped 1 previous similar message [ 6601.803025] Lustre: server umount lustre-OST0000 complete [ 6608.354116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6608.355651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6608.361048] Lustre: Skipped 2 previous similar messages [ 6608.361496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6608.378969] Lustre: Skipped 1 previous similar message [ 6613.472686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6613.476853] Lustre: Skipped 1 previous similar message [ 6618.079118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6618.154787] Lustre: server umount lustre-MDT0000 complete [ 6618.592699] LustreError: 272106:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6618.603700] LustreError: 272106:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 6620.105684] LustreError: 272844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752539 with bad export cookie 9343042544892685633 [ 6620.112196] LustreError: 272844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6620.112270] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6620.306672] Lustre: server umount lustre-MDT0001 complete [ 6625.763151] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6627.376359] Key type lgssc unregistered [ 6627.577811] LNet: 276413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6627.588315] LNetError: 276413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6627.600655] LNet: Removed LNI 192.168.206.151@tcp [ 6628.106465] Key type .llcrypt unregistered [ 6628.108166] Key type ._llcrypt unregistered [ 6638.238218] Key type ._llcrypt registered [ 6638.240961] Key type .llcrypt registered [ 6638.307377] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6638.713305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6638.772063] alg: No test for adler32 (adler32-zlib) [ 6639.669651] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 6639.810195] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6641.431199] Key type lgssc registered [ 6641.974369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6646.007369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6646.019936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.247232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6649.117059] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6653.544803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6653.761408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6656.032335] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6660.068496] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6663.051555] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6666.998830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6667.220300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6670.569062] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6672.363969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6674.696751] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6687.207333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6687.218568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6687.230082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6688.741722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6688.743487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6688.745988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6691.325159] Lustre: server umount lustre-OST0000 complete [ 6693.856472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6693.859396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6693.868144] Lustre: Skipped 1 previous similar message [ 6693.875126] Lustre: Skipped 2 previous similar messages [ 6698.976916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6698.980901] Lustre: Skipped 1 previous similar message [ 6699.777211] Lustre: server umount lustre-MDT0000 complete [ 6701.894425] LustreError: 277588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752620 with bad export cookie 7283829857307970723 [ 6701.900180] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6701.901364] LustreError: 277588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6702.089873] Lustre: server umount lustre-MDT0001 complete [ 6707.998362] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6709.499639] Key type lgssc unregistered [ 6709.667583] LNet: 281316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6709.671802] LNetError: 281316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6710.693851] LNet: Removed LNI 192.168.206.151@tcp [ 6711.172191] Key type .llcrypt unregistered [ 6711.174479] Key type ._llcrypt unregistered [ 6717.535782] Key type ._llcrypt registered [ 6717.540747] Key type .llcrypt registered [ 6717.605320] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6726.877830] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 6735.888710] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6736.379341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6736.425698] alg: No test for adler32 (adler32-zlib) [ 6737.376501] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 6737.539841] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6739.167224] Key type lgssc registered [ 6739.809506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6758.380652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6758.399939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.511752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6759.531970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6759.581195] Lustre: lustre-MDT0000: new disk, initializing [ 6759.647904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6759.657568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6762.005483] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6768.457326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6768.530741] Lustre: 284839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6768.567769] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6768.570623] Lustre: Skipped 1 previous similar message [ 6768.611689] Lustre: lustre-MDT0001: new disk, initializing [ 6768.665644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6768.685838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6768.694078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6771.055366] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6776.943088] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6780.035387] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6783.604508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6783.766408] Lustre: lustre-OST0000: new disk, initializing [ 6783.768370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6783.771709] Lustre: 286808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6783.816407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6785.422746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6785.429166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6785.490283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6786.733909] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6792.306561] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6794.756330] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6794.918536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6797.266731] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6797.377784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6800.865743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6800.871075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.872104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6800.878342] Lustre: Skipped 1 previous similar message [ 6800.886410] Lustre: Skipped 1 previous similar message [ 6804.587152] Lustre: server umount lustre-OST0000 complete [ 6809.569688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6809.576193] LustreError: Skipped 1 previous similar message [ 6809.578485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6809.592316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6811.105033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6811.105020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6812.814334] Lustre: server umount lustre-MDT0000 complete [ 6814.873380] LustreError: 284832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752733 with bad export cookie 2538555668728889600 [ 6814.876463] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6814.885321] LustreError: 284832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6815.109844] Lustre: server umount lustre-MDT0001 complete [ 6819.782623] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:45:39 (1788752739) [ 6823.212882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6823.480382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6825.668087] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6828.515772] LustreError: 289220:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6828.526327] LustreError: 289220:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 6830.285702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6830.498772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6832.963188] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6837.193580] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6840.342385] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6844.127584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6844.317336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6847.773212] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6851.382220] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6860.774744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6860.775898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6860.785219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6860.792494] Lustre: Skipped 2 previous similar messages [ 6862.059644] Lustre: server umount lustre-OST0000 complete [ 6863.900152] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6863.997863] Lustre: server umount lustre-MDT0000 complete [ 6865.888860] LustreError: 289219:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6865.889808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6865.910130] Lustre: Skipped 2 previous similar messages [ 6865.950023] LustreError: 289199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752785 with bad export cookie 2538555668728890482 [ 6865.952580] LustreError: 289954:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 6865.952802] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6865.955202] LustreError: 289199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6866.109097] Lustre: server umount lustre-MDT0001 complete [ 6871.637584] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6873.344906] Key type lgssc unregistered [ 6873.526518] LNet: 292748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6873.533503] LNetError: 292748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6873.551048] LNet: Removed LNI 192.168.206.151@tcp [ 6874.072942] Key type .llcrypt unregistered [ 6874.074555] Key type ._llcrypt unregistered [ 6888.458402] Key type ._llcrypt registered [ 6888.460037] Key type .llcrypt registered [ 6888.510241] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 6896.495500] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6896.909685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6896.940383] alg: No test for adler32 (adler32-zlib) [ 6897.875803] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 6898.015596] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6899.632474] Key type lgssc registered [ 6900.241543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6921.837868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6921.846773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.953959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6922.971180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6923.013799] Lustre: lustre-MDT0000: new disk, initializing [ 6923.050462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6923.066359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6925.269754] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6930.842384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.895260] Lustre: 296541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6930.938152] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6930.941153] Lustre: Skipped 1 previous similar message [ 6930.987887] Lustre: lustre-MDT0001: new disk, initializing [ 6931.042564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6931.064103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6931.068698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6932.971828] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6938.069758] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6940.894359] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6944.208821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6944.352908] Lustre: lustre-OST2710: new disk, initializing [ 6944.356566] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6944.361543] Lustre: 298509:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6944.407073] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6946.384408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6946.390632] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6946.446517] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6947.372542] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6952.451209] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6956.108482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6956.228862] Lustre: lustre-OST03e8: new disk, initializing [ 6956.231947] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6956.237603] Lustre: 299573:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6956.298201] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6959.491219] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6963.383983] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6964.247884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6964.257636] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6964.322676] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6967.023578] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6967.128573] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6969.368483] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6969.475608] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6971.745983] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6971.876210] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6973.987845] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6974.086796] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6976.992515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6977.000785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6977.010573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6979.557335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6979.557388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6979.564696] Lustre: Skipped 1 previous similar message [ 6979.572897] Lustre: Skipped 3 previous similar messages [ 6981.890636] Lustre: server umount lustre-MDT0000 complete [ 6983.601832] LustreError: 297336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788752902 with bad export cookie 4639864890621185340 [ 6983.602201] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6983.609280] LustreError: 297336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6983.769631] Lustre: server umount lustre-MDT0001 complete [ 6995.883513] Lustre: server umount lustre-OST2710 complete [ 7007.164162] Lustre: server umount lustre-OST03e8 complete [ 7017.386632] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 7022.570843] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7043.023219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7043.129515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7043.144589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7043.196120] Lustre: lustre-MDT0000: new disk, initializing [ 7043.234868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7043.246215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7045.532666] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7051.375060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7051.435296] Lustre: 304812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7051.465687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7051.468553] Lustre: Skipped 1 previous similar message [ 7051.508512] Lustre: lustre-MDT0001: new disk, initializing [ 7051.555680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7051.573299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7051.582393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7053.538349] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7058.402451] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7061.021300] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7064.397545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7064.550653] Lustre: 306781:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7067.432748] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7072.978910] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7074.333463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7074.338873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7074.399809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7075.439673] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7075.578045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7077.860934] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7077.975470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7079.393742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7079.395254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7079.400197] LustreError: Skipped 1 previous similar message [ 7079.401195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7079.418806] Lustre: Skipped 2 previous similar messages [ 7081.006631] Lustre: server umount lustre-OST0000 complete [ 7084.513472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7084.515132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7084.519638] Lustre: Skipped 1 previous similar message [ 7084.523345] Lustre: Skipped 1 previous similar message [ 7089.633273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7089.638129] Lustre: Skipped 1 previous similar message [ 7097.311886] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7097.459080] Lustre: server umount lustre-MDT0000 complete [ 7099.531258] LustreError: 306780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753018 with bad export cookie 4639864890621187020 [ 7099.535754] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7099.537419] LustreError: 306780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7099.702582] Lustre: server umount lustre-MDT0001 complete [ 7104.402811] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:50:23 (1788753023) [ 7105.257401] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7106.316854] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:50:25 (1788753025) [ 7108.210944] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7122.869346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7123.006888] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7123.021190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7123.097389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7123.100362] Lustre: Skipped 1 previous similar message [ 7125.272189] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7129.501795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7129.571871] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7129.589517] Lustre: 310633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7131.767798] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7149.349767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7149.450804] Lustre: Denying initial registration attempt from nid 192.168.206.151@tcp, specified as failover [ 7149.457179] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7149.464850] LustreError: 312084:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 7149.471419] LustreError: 312084:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7149.475161] LustreError: 312084:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7149.508569] Lustre: server umount lustre-OST0000 complete [ 7149.510588] LustreError: 312084:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7155.680680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7155.686590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7155.692193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7155.694543] Lustre: Skipped 2 previous similar messages [ 7158.823563] Lustre: server umount lustre-MDT0000 complete [ 7160.701748] LustreError: 310626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753079 with bad export cookie 4639864890621187888 [ 7160.704237] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7160.707482] LustreError: 310626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7160.882617] Lustre: server umount lustre-MDT0001 complete [ 7165.426450] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7166.717181] Key type lgssc unregistered [ 7166.873570] LNet: 313203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7166.878627] LNetError: 313203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7166.888523] LNet: Removed LNI 192.168.206.151@tcp [ 7167.280782] Key type .llcrypt unregistered [ 7167.282308] Key type ._llcrypt unregistered [ 7172.019658] Key type ._llcrypt registered [ 7172.021199] Key type .llcrypt registered [ 7172.072323] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7172.455800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7172.525060] alg: No test for adler32 (adler32-zlib) [ 7173.408832] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 7173.536583] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7175.135387] Key type lgssc registered [ 7175.608524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7202.782447] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7206.239715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7206.255173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.382699] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7207.392526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7207.464697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7209.246703] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7213.076220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7213.129626] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7213.145474] Lustre: 315937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7213.280774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7215.310882] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7218.695105] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7221.209521] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7236.912689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7237.012080] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7237.110542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7239.575857] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7242.521099] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7243.382167] Lustre: server umount lustre-OST0000 complete [ 7246.838201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7246.838964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7246.845330] Lustre: Skipped 1 previous similar message [ 7246.849683] Lustre: Skipped 1 previous similar message [ 7251.098571] Lustre: server umount lustre-MDT0000 complete [ 7252.611304] LustreError: 317800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753171 with bad export cookie 13599545772931055482 [ 7252.614439] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7252.615113] LustreError: 317800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7252.735788] Lustre: server umount lustre-MDT0001 complete [ 7256.901743] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7258.094241] Key type lgssc unregistered [ 7258.232423] LNet: 319498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7258.235675] LNetError: 319498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7258.243273] LNet: Removed LNI 192.168.206.151@tcp [ 7258.599113] Key type .llcrypt unregistered [ 7258.600929] Key type ._llcrypt unregistered [ 7269.465390] Key type ._llcrypt registered [ 7269.466938] Key type .llcrypt registered [ 7269.517889] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7270.016788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7270.025447] alg: No test for adler32 (adler32-zlib) [ 7270.919731] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 7271.022990] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7272.616626] Key type lgssc registered [ 7273.096177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7276.428382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7276.437491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7277.613346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7279.233807] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7282.674252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.825888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7284.424246] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7287.478489] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7289.705976] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7294.566529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7294.760349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7297.326908] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7300.536944] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7317.202650] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 7317.474360] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7317.479978] Lustre: Skipped 2 previous similar messages [ 7320.988405] Lustre: server umount lustre-MDT0000 complete [ 7322.312340] LustreError: 320883:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7322.320312] LustreError: 320883:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7322.496444] LustreError: 320914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753241 with bad export cookie 12229221354443447675 [ 7322.498826] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7322.503901] LustreError: 320914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7322.592870] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7322.601135] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7322.603693] Lustre: Skipped 3 previous similar messages [ 7327.432305] LustreError: 320884:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7327.439017] LustreError: 320884:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7327.442632] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 7328.703235] Lustre: server umount lustre-MDT0001 complete [ 7331.571732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7335.461668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7347.168245] LustreError: 320198:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000024a566b3 x1875643664982912/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7347.341544] LustreError: 322717:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7347.351654] LustreError: 322717:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7347.379516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7347.416051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 7349.350845] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7352.481035] LustreError: 324499:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7352.493083] LustreError: 324499:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7353.223678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7353.394393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7355.264592] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7357.480220] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7358.806546] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7363.643311] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 7366.510286] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7368.311937] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 7372.774363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7372.778379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7372.780111] Lustre: Skipped 1 previous similar message [ 7372.789986] Lustre: Skipped 2 previous similar messages [ 7376.491489] Lustre: server umount lustre-OST0000 complete [ 7378.912168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7378.915571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7378.923551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7383.009520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7384.415728] Lustre: server umount lustre-MDT0000 complete [ 7385.929506] LustreError: 320870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753305 with bad export cookie 12229221354443454801 [ 7385.932400] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7385.936380] LustreError: 320870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7386.064459] Lustre: server umount lustre-MDT0001 complete [ 7390.885350] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7392.269976] Key type lgssc unregistered [ 7392.437785] LNet: 327167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7392.441650] LNetError: 327167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7392.457548] LNet: Removed LNI 192.168.206.151@tcp [ 7392.846779] Key type .llcrypt unregistered [ 7392.849386] Key type ._llcrypt unregistered [ 7405.358046] Key type ._llcrypt registered [ 7405.359361] Key type .llcrypt registered [ 7405.421214] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7405.962505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7405.981976] alg: No test for adler32 (adler32-zlib) [ 7406.917368] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 7407.048800] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7408.655185] Key type lgssc registered [ 7409.306835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7413.371227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7413.381460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7414.619534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7417.043480] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7421.745789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7421.953636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7424.080790] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7427.782658] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7430.474953] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7432.164395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7432.171595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.179881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7433.186839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7433.188260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.206676] Lustre: server umount lustre-MDT0000 complete [ 7438.305566] LustreError: 328560:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7438.311724] LustreError: 328560:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7439.916625] LustreError: 328540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753359 with bad export cookie 8818827933111735052 [ 7439.918636] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7439.921564] LustreError: 328540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7440.073580] Lustre: server umount lustre-MDT0001 complete [ 7447.180515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.344730] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7447.373288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7447.486376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7449.576266] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7453.522819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7453.595465] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7453.623308] Lustre: 331142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7453.753166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7455.600634] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7459.053913] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7461.694432] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7467.108044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7467.183800] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7467.188434] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7467.193045] LustreError: 333026:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 7467.196426] LustreError: 333026:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7467.198823] LustreError: 333026:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7467.226478] Lustre: server umount lustre-OST0000 complete [ 7467.228818] LustreError: 333026:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7472.612192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7472.705820] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7472.781219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7475.131571] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7478.153564] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7481.376383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7481.415983] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7481.458251] Lustre: lustre-OST0001: new disk, initializing [ 7481.461377] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7481.465981] Lustre: 334555:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7481.505518] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7481.819485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7481.825351] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7481.844211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 7481.875799] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7483.956238] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7486.824643] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7492.067344] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7492.068295] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7492.071382] LustreError: Skipped 1 previous similar message [ 7492.077906] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7492.081979] Lustre: Skipped 2 previous similar messages [ 7497.187563] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7497.190360] Lustre: Skipped 1 previous similar message [ 7502.305698] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7502.310752] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7502.362258] Lustre: server umount lustre-OST0001 complete [ 7507.425354] LustreError: 334611:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7507.428493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7507.429730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7507.429739] Lustre: Skipped 1 previous similar message [ 7507.441035] LustreError: 334611:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7507.442391] LustreError: Skipped 1 previous similar message [ 7512.544810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7512.548719] Lustre: Skipped 3 previous similar messages [ 7512.551162] LustreError: 334614:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7512.558162] LustreError: 334614:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7517.668222] LustreError: 333490:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7517.674589] LustreError: 333490:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7518.175160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7518.259047] Lustre: server umount lustre-OST0000 complete [ 7520.224144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7520.228785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.234590] Lustre: Skipped 1 previous similar message [ 7526.151457] Lustre: server umount lustre-MDT0000 complete [ 7527.907452] LustreError: 331154:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7527.914125] LustreError: 331154:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 7528.056740] LustreError: 331882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753447 with bad export cookie 8818827933111735605 [ 7528.057094] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7528.060819] LustreError: 331882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7528.209898] Lustre: server umount lustre-MDT0001 complete [ 7532.804353] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7533.947632] Key type lgssc unregistered [ 7534.081549] LNet: 336521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7534.085136] LNetError: 336521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7534.096516] LNet: Removed LNI 192.168.206.151@tcp [ 7534.466599] Key type .llcrypt unregistered [ 7534.468067] Key type ._llcrypt unregistered [ 7570.498631] Key type ._llcrypt registered [ 7570.500093] Key type .llcrypt registered [ 7570.549994] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 7577.696204] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7578.099917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7578.152179] alg: No test for adler32 (adler32-zlib) [ 7579.047443] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 7579.177141] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7580.791287] Key type lgssc registered [ 7581.230193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7596.109705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7596.117266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7597.221556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7597.239469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7597.282978] Lustre: lustre-MDT0000: new disk, initializing [ 7597.317132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7597.325569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7599.088141] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7604.090347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7604.143304] Lustre: 341417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7604.173228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7604.175468] Lustre: Skipped 1 previous similar message [ 7604.214408] Lustre: lustre-MDT0001: new disk, initializing [ 7604.256382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7604.273722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7604.278901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7605.951768] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7610.502919] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7612.947373] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7616.120674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7616.267694] Lustre: lustre-OST0000: new disk, initializing [ 7616.270818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7616.276507] Lustre: 343387:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7616.324388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.331340] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7624.543616] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7625.759879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7625.766875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7625.834826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7626.668564] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7626.769481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7628.676314] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7628.797261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7630.817635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7630.818388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7630.828695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7630.833110] Lustre: Skipped 1 previous similar message [ 7635.815418] Lustre: server umount lustre-OST0000 complete [ 7640.033993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7640.040789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7640.048305] Lustre: Skipped 1 previous similar message [ 7640.051391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7641.057073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7641.057523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7643.639215] Lustre: server umount lustre-MDT0000 complete [ 7645.297326] LustreError: 341410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753564 with bad export cookie 8441706401442580650 [ 7645.300449] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7645.304884] LustreError: 341410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7645.472460] Lustre: server umount lustre-MDT0001 complete [ 7649.134309] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:59:28 (1788753568) [ 7652.251347] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:59:32 (1788753572) [ 7655.672175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7655.881367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7657.485610] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7661.025851] LustreError: 346138:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7661.032561] LustreError: 346138:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7661.097810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7661.295660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7663.060398] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7666.205316] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7668.879194] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7671.909239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7672.072103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7674.977966] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7678.666653] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7693.906609] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7694.650719] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7695.324437] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7696.093967] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7696.996659] Lustre: Failing over lustre-MDT0000 [ 7697.164847] Lustre: server umount lustre-MDT0000 complete [ 7697.376116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7697.379917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7697.387085] LustreError: 346138:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7701.717394] LustreError: 346133:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7701.989450] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7701.997897] Lustre: Skipped 1 previous similar message [ 7706.826057] LustreError: 348637:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7706.837784] LustreError: 348637:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 7711.276637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7711.374985] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7711.506239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7711.538386] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7711.948893] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7713.416789] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7716.837444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7716.850231] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7716.870395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7719.348270] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7720.070451] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7722.283506] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7723.124444] Lustre: Failing over lustre-MDT0000 [ 7723.243559] Lustre: server umount lustre-MDT0000 complete [ 7727.075135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7727.075325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7727.076511] LustreError: 348637:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7727.076520] LustreError: 348637:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 7727.095701] Lustre: Skipped 1 previous similar message [ 7737.096295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7737.164493] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7737.317598] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7737.546954] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7739.373412] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7742.444632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7742.450895] Lustre: Skipped 2 previous similar messages [ 7742.461681] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7742.519090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7745.665162] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7746.488130] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7752.677718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7752.679976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7752.680967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7752.680971] Lustre: Skipped 1 previous similar message [ 7752.684880] Lustre: Skipped 2 previous similar messages [ 7755.673213] Lustre: server umount lustre-MDT0000 complete [ 7757.306795] LustreError: 346120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753676 with bad export cookie 8441706401442583331 [ 7757.309257] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7757.314334] LustreError: 346120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7757.465452] Lustre: server umount lustre-MDT0001 complete [ 7769.459945] Lustre: server umount lustre-OST0000 complete [ 7776.118606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7776.320685] LustreError: 352505:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7776.330914] LustreError: 352505:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 7 previous similar messages [ 7776.362605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7776.366909] Lustre: Skipped 1 previous similar message [ 7777.965397] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7781.473878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7783.550535] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7786.712686] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7788.877981] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7791.920236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7794.641140] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7797.562979] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7801.317824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7811.552523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7811.556196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7811.562572] Lustre: Skipped 1 previous similar message [ 7811.564976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7811.567405] Lustre: Skipped 2 previous similar messages [ 7816.083910] Lustre: server umount lustre-OST0000 complete [ 7822.817178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7822.821068] Lustre: Skipped 5 previous similar messages [ 7823.586542] Lustre: server umount lustre-MDT0000 complete [ 7825.040376] LustreError: 353240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753744 with bad export cookie 8441706401442584080 [ 7825.043935] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7825.047039] LustreError: 353240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7825.178873] Lustre: server umount lustre-MDT0001 complete [ 7829.307598] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7830.536610] Key type lgssc unregistered [ 7830.680395] LNet: 356055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7830.683974] LNetError: 356055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7830.692429] LNet: Removed LNI 192.168.206.151@tcp [ 7831.040089] Key type .llcrypt unregistered [ 7831.041673] Key type ._llcrypt unregistered [ 7843.167117] Key type ._llcrypt registered [ 7843.168768] Key type .llcrypt registered [ 7843.221833] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7843.662946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7843.721855] alg: No test for adler32 (adler32-zlib) [ 7844.608765] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 7844.716087] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7846.311646] Key type lgssc registered [ 7846.796480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7849.709228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7849.716212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.893774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7852.471627] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7855.822015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7855.979260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7857.978527] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7861.371361] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7863.867330] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7867.097187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7867.257321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7869.689493] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7872.949772] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7875.433884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7891.936842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7891.938315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7891.939864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7891.944187] Lustre: Skipped 1 previous similar message [ 7893.596354] Lustre: server umount lustre-MDT0000 complete [ 7894.937434] LustreError: 357525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753814 with bad export cookie 15926090859633328885 [ 7894.941286] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7894.941549] LustreError: 357525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7895.065756] Lustre: server umount lustre-MDT0001 complete [ 7906.719190] Lustre: server umount lustre-OST0000 complete [ 7911.549891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7911.752198] LustreError: 361271:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7911.761272] LustreError: 361271:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7911.781611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7913.520554] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7917.026595] LustreError: 361272:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7917.049887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7917.220989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7918.885958] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7922.093756] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7924.374975] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7927.042887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7927.170828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7929.524567] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7932.438414] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7936.423834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7946.721586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7946.722371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7946.729203] Lustre: Skipped 2 previous similar messages [ 7946.734486] Lustre: Skipped 4 previous similar messages [ 7951.027266] Lustre: server umount lustre-MDT0000 complete [ 7951.840956] LustreError: 361271:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7951.846538] LustreError: 361271:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 7952.532800] LustreError: 363113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753871 with bad export cookie 15926090859633329984 [ 7952.535170] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7952.537735] LustreError: 363113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7952.663301] Lustre: server umount lustre-MDT0001 complete [ 7964.449826] Lustre: server umount lustre-OST0000 complete [ 7970.547775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7970.751206] LustreError: 365178:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7970.780692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7972.449974] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7975.753260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7977.584463] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7980.511255] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7982.622847] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7985.173929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7987.553868] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7990.439224] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7995.559108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8000.484326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8000.485953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8000.493687] Lustre: Skipped 1 previous similar message [ 8003.563709] Lustre: server umount lustre-OST0000 complete [ 8005.600184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8005.605405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8005.607640] Lustre: Skipped 2 previous similar messages [ 8010.720835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8010.724515] Lustre: Skipped 1 previous similar message [ 8011.177345] Lustre: server umount lustre-MDT0000 complete [ 8012.751229] LustreError: 367957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788753931 with bad export cookie 15926090859633330992 [ 8012.753490] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8012.755502] LustreError: 367957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8012.877688] Lustre: server umount lustre-MDT0001 complete [ 8017.269236] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8018.445276] Key type lgssc unregistered [ 8018.595539] LNet: 368707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8018.598552] LNetError: 368707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8018.608508] LNet: Removed LNI 192.168.206.151@tcp [ 8018.973123] Key type .llcrypt unregistered [ 8018.974954] Key type ._llcrypt unregistered [ 8029.587822] Key type ._llcrypt registered [ 8029.588989] Key type .llcrypt registered [ 8029.628936] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8029.972494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8030.003218] alg: No test for adler32 (adler32-zlib) [ 8030.866035] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 8030.964661] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8032.551173] Key type lgssc registered [ 8032.951830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8035.826912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8035.835869] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8035.839996] LustreError: 370122:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8035.852941] LustreError: 370122:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8035.858311] LustreError: 370122:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8035.863220] LustreError: 370122:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8035.868284] LustreError: 370122:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8039.264122] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8039.269738] LustreError: 370519:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8039.282835] LustreError: 370519:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8039.287201] LustreError: 370519:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8039.294332] LustreError: 370519:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8039.299265] LustreError: 370519:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8047.568870] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8048.852395] Key type lgssc unregistered [ 8049.005134] LNet: 371519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8049.009292] LNetError: 371519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8049.020676] LNet: Removed LNI 192.168.206.151@tcp [ 8049.430378] Key type .llcrypt unregistered [ 8049.432053] Key type ._llcrypt unregistered [ 8056.410441] Key type ._llcrypt registered [ 8056.411788] Key type .llcrypt registered [ 8056.451976] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8063.478282] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8063.849790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8063.906663] alg: No test for adler32 (adler32-zlib) [ 8064.774147] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 8064.870337] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8066.455200] Key type lgssc registered [ 8066.874405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8081.471062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8081.478416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8082.576309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8082.590930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8082.629844] Lustre: lustre-MDT0000: new disk, initializing [ 8082.658771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8082.666311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8084.284548] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8089.168944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8089.217443] Lustre: 374647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8089.247620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8089.250726] Lustre: Skipped 1 previous similar message [ 8089.288269] Lustre: lustre-MDT0001: new disk, initializing [ 8089.328342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8089.345176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8089.351342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8090.989897] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8095.781752] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8098.283782] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8101.157370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8101.271690] Lustre: lustre-OST0000: new disk, initializing [ 8101.273905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8101.277846] Lustre: 376615:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8101.315539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8103.564942] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8107.862866] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8109.062470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8109.066912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8109.114196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8109.597287] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8109.694789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8111.345479] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8111.438972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8114.146159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8114.146971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.149801] LustreError: Skipped 1 previous similar message [ 8114.151141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8114.155557] Lustre: Skipped 1 previous similar message [ 8118.375138] Lustre: server umount lustre-OST0000 complete [ 8120.288847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8120.294509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8120.302232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8120.305134] Lustre: Skipped 1 previous similar message [ 8124.383984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8124.385371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8126.241668] Lustre: server umount lustre-MDT0000 complete [ 8127.950460] LustreError: 374640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754047 with bad export cookie 16157526043257844896 [ 8127.953647] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8127.957132] LustreError: 374640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8128.081711] Lustre: server umount lustre-MDT0001 complete [ 8131.748146] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8132.509776] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:07:32 (1788754052) [ 8135.385864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8135.660411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8137.354540] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8140.769334] LustreError: 379070:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8140.776267] LustreError: 379070:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 8140.921901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8141.067831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8142.759426] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8145.587967] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8147.831674] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8150.526929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8150.661356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8152.892348] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8155.840873] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8158.729926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8158.812438] Lustre: lustre-OST0001: new disk, initializing [ 8158.815776] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8158.820630] Lustre: 381902:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8161.330366] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8162.577638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8162.581935] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8162.615340] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8165.840379] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8172.245580] Lustre: lustre-OST0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 8172.248943] Lustre: Skipped 1 previous similar message [ 8173.031812] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8173.032318] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8173.038973] Lustre: Skipped 1 previous similar message [ 8173.044516] LustreError: Skipped 1 previous similar message [ 8174.100792] Lustre: server umount lustre-OST0001 complete [ 8178.149295] LustreError: 380889:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8178.156939] LustreError: 380889:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 8183.269048] LustreError: 380891:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8183.274624] LustreError: 380891:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 8187.873878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8187.877474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.884681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8187.888597] Lustre: Skipped 2 previous similar messages [ 8188.389856] LustreError: 380892:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8188.397457] LustreError: 380892:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 8192.790139] Lustre: server umount lustre-OST0000 complete [ 8197.599860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8197.604791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8197.611154] Lustre: Skipped 1 previous similar message [ 8197.615571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8197.618625] Lustre: Skipped 2 previous similar messages [ 8200.545865] Lustre: server umount lustre-MDT0000 complete [ 8202.213827] LustreError: 379804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754121 with bad export cookie 16157526043257845764 [ 8202.214237] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8202.221723] LustreError: 379804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8208.382857] Lustre: server umount lustre-MDT0001 complete [ 8212.789339] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8213.904028] Key type lgssc unregistered [ 8214.045669] LNet: 383884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8214.049458] LNetError: 383884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8214.057402] LNet: Removed LNI 192.168.206.151@tcp [ 8214.417613] Key type .llcrypt unregistered [ 8214.419061] Key type ._llcrypt unregistered [ 8235.774228] Key type ._llcrypt registered [ 8235.775622] Key type .llcrypt registered [ 8235.824730] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8236.245669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8236.281761] alg: No test for adler32 (adler32-zlib) [ 8237.153651] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 8237.264379] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8238.855271] Key type lgssc registered [ 8239.279763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8242.223855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8242.232077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8243.332242] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8243.339917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8243.394130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8244.783508] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8247.799026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8247.841522] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8247.855418] Lustre: 386655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8247.972071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8249.438742] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8251.970298] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8253.854900] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8254.944627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.945534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8254.949560] Lustre: Skipped 1 previous similar message [ 8260.064105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8260.066280] Lustre: Skipped 2 previous similar messages [ 8260.761341] Lustre: server umount lustre-MDT0000 complete [ 8262.077141] LustreError: 386645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754181 with bad export cookie 13411073949792838532 [ 8262.080664] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8262.081275] LustreError: 386645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8262.192337] Lustre: server umount lustre-MDT0001 complete [ 8265.883080] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:09:45 (1788754185) [ 8268.462421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8268.650070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8270.218231] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8273.397250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8273.544463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8275.090903] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8277.746657] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8279.746936] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8282.271800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8282.352733] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8282.421284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8284.462897] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8287.338975] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8292.457094] Lustre: Permanently deactivating lustre-OST0000 [ 8292.463785] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8295.216969] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8295.831978] LustreError: 392055:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8295.835923] LustreError: 392055:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8295.839217] LustreError: 392055:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8296.491924] LustreError: 392104:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8296.494867] LustreError: 392104:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8303.316612] Lustre: server umount lustre-OST0000 complete [ 8304.319028] LustreError: 392353:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8304.322979] LustreError: 392353:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8308.704593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8308.705198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8308.711171] Lustre: Skipped 1 previous similar message [ 8313.824739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8313.827920] Lustre: Skipped 2 previous similar messages [ 8315.909596] Lustre: server umount lustre-MDT0000 complete [ 8317.437554] LustreError: 391000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754236 with bad export cookie 13411073949792839099 [ 8317.439565] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8317.442948] LustreError: 391000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8327.719589] Lustre: server umount lustre-MDT0001 complete [ 8331.401635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8333.033966] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8334.072389] LustreError: 393511:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8336.570310] Lustre: 393850:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.151@tcp [ 8337.360412] Lustre: server umount MGS complete [ 8341.873177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8342.074597] LustreError: 394572:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8342.080732] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8342.082764] LustreError: 394572:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 8342.085421] Lustre: Skipped 1 previous similar message [ 8342.104695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8343.710179] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8347.104086] LustreError: 394573:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8347.268849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.404337] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8349.084862] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8352.198243] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8354.695967] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8355.546829] Lustre: Permanently reactivating lustre-OST0000 [ 8355.554177] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8355.557074] Lustre: Skipped 2 previous similar messages [ 8379.359145] LustreError: 396304:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8383.405590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8383.550832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8383.553542] Lustre: Skipped 1 previous similar message [ 8385.976670] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8388.913854] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8405.472608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8405.474287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8405.474934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8405.485809] Lustre: Skipped 1 previous similar message [ 8407.191647] Lustre: server umount lustre-OST0000 complete [ 8409.056311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8409.062080] LustreError: Skipped 1 previous similar message [ 8409.064424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8415.712256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8415.715253] Lustre: Skipped 5 previous similar messages [ 8423.391150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8423.455194] Lustre: server umount lustre-MDT0000 complete [ 8424.833164] LustreError: 395308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754343 with bad export cookie 13411073949792840121 [ 8424.836230] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8424.838510] LustreError: 395308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8424.975306] Lustre: server umount lustre-MDT0001 complete [ 8429.174568] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8430.370126] Key type lgssc unregistered [ 8430.504433] LNet: 399147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8430.507735] LNetError: 399147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8430.514333] LNet: Removed LNI 192.168.206.151@tcp [ 8430.874084] Key type .llcrypt unregistered [ 8430.875724] Key type ._llcrypt unregistered [ 8437.903979] Key type ._llcrypt registered [ 8437.905453] Key type .llcrypt registered [ 8437.942459] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8444.742442] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8445.066477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8445.087723] alg: No test for adler32 (adler32-zlib) [ 8445.966727] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 8446.075034] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8447.671190] Key type lgssc registered [ 8448.059038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8460.728542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8460.734738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8461.829387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8461.844484] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8461.884825] Lustre: lustre-MDT0000: new disk, initializing [ 8461.906855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8461.914036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8463.490945] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8468.017502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8468.069277] Lustre: 402273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8468.096613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8468.099782] Lustre: Skipped 1 previous similar message [ 8468.126629] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8468.141345] Lustre: lustre-MDT0001: new disk, initializing [ 8468.177654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8468.193564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8468.197479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8469.793653] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8473.588269] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8475.521566] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8477.833641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8477.947123] Lustre: lustre-OST0000: new disk, initializing [ 8477.949375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8477.953517] Lustre: 404241:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8477.990059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8480.028663] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8481.464502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8481.468197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8481.500784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8483.757695] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8485.314593] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8485.399684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8486.975566] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8487.062360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8492.001671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8492.002014] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8492.003285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8492.006312] Lustre: Skipped 1 previous similar message [ 8492.008551] LustreError: Skipped 1 previous similar message [ 8497.120939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8497.123075] Lustre: Skipped 2 previous similar messages [ 8502.239112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8502.242206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8502.244811] Lustre: Skipped 1 previous similar message [ 8502.301983] Lustre: server umount lustre-OST0000 complete [ 8504.287741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8504.290524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8507.360043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8507.360762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8507.362135] Lustre: Skipped 1 previous similar message [ 8509.787550] Lustre: server umount lustre-MDT0000 complete [ 8511.057209] LustreError: 403066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754430 with bad export cookie 17381117687275546839 [ 8511.059799] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8511.061056] LustreError: 403066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8511.178191] Lustre: server umount lustre-MDT0001 complete [ 8514.170934] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8514.817922] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:13:54 (1788754434) [ 8517.340599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8517.569420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8519.214489] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8522.427112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8522.608758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8524.383445] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8527.690548] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8529.976225] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8533.053579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8533.213560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8535.681224] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8538.723307] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8539.583548] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8546.582669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8558.051172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8558.051966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8558.054885] LustreError: Skipped 1 previous similar message [ 8558.062182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8558.067076] Lustre: Skipped 1 previous similar message [ 8559.568959] Lustre: server umount lustre-OST0000 complete [ 8563.168979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8563.175821] Lustre: Skipped 2 previous similar messages [ 8567.403474] Lustre: server umount lustre-MDT0000 complete [ 8568.288377] LustreError: 407426:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8568.297711] LustreError: 407426:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 3 previous similar messages [ 8569.083529] LustreError: 406680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754488 with bad export cookie 17381117687275547721 [ 8569.085078] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8569.088188] LustreError: 406680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8569.222092] Lustre: server umount lustre-MDT0001 complete [ 8573.768650] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8575.044427] Key type lgssc unregistered [ 8575.184602] LNet: 410444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8575.188082] LNetError: 410444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8575.195418] LNet: Removed LNI 192.168.206.151@tcp [ 8575.559558] Key type .llcrypt unregistered [ 8575.561302] Key type ._llcrypt unregistered [ 8585.175864] Key type ._llcrypt registered [ 8585.178118] Key type .llcrypt registered [ 8585.223346] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8585.368483] Key type .llcrypt unregistered [ 8585.369936] Key type ._llcrypt unregistered [ 8593.239586] Key type ._llcrypt registered [ 8593.240853] Key type .llcrypt registered [ 8593.292395] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8593.832956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8593.851051] alg: No test for adler32 (adler32-zlib) [ 8594.724842] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 8594.834590] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8596.423292] Key type lgssc registered [ 8596.855338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8599.993871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8600.004414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8601.185633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8602.808292] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8606.324063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8606.456781] LustreError: 412617:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8606.460479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8606.465669] LustreError: 412617:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 8608.817878] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8611.549505] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8611.618588] LustreError: 412623:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8611.623908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8611.626743] LustreError: 412623:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 8614.290414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8614.490439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8616.197125] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8624.608493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8624.612101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.615698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8624.617741] Lustre: Skipped 1 previous similar message [ 8629.734178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.737710] Lustre: Skipped 1 previous similar message [ 8629.984320] Lustre: server umount lustre-OST0000 complete [ 8633.828451] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8633.833837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8633.840253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8633.843027] Lustre: Skipped 1 previous similar message [ 8634.849101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8637.596184] Lustre: server umount lustre-MDT0000 complete [ 8639.153971] LustreError: 412605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754558 with bad export cookie 2790656483518171596 [ 8639.157611] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8639.159582] LustreError: 412605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8639.311396] Lustre: server umount lustre-MDT0001 complete [ 8643.598829] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8644.737301] Key type lgssc unregistered [ 8644.887532] LNet: 415767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8644.891979] LNetError: 415767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8644.903450] LNet: Removed LNI 192.168.206.151@tcp [ 8645.246523] Key type .llcrypt unregistered [ 8645.248113] Key type ._llcrypt unregistered [ 8655.807181] Key type ._llcrypt registered [ 8655.808635] Key type .llcrypt registered [ 8655.856714] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8656.365996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8656.373704] alg: No test for adler32 (adler32-zlib) [ 8657.265133] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 8657.376083] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8658.975198] Key type lgssc registered [ 8659.469201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8662.566261] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8662.573826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8678.047332] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8683.167302] LustreError: 417129:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.151@tcp: failed processing log, type 4: rc = -110 [ 8709.791251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8712.052545] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8714.923980] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8717.605991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8717.812268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8717.840960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8719.361170] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8722.668735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8722.862649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8724.594361] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8727.514447] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8729.643053] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8733.152600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8733.153419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8733.157090] LustreError: Skipped 1 previous similar message [ 8733.162434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8733.165687] Lustre: Skipped 1 previous similar message [ 8735.203039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8735.205604] Lustre: Skipped 1 previous similar message [ 8738.400415] Lustre: server umount lustre-OST0000 complete [ 8740.322342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8740.324237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8740.331757] Lustre: Skipped 2 previous similar messages [ 8745.440718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8745.443213] Lustre: Skipped 2 previous similar messages [ 8746.015260] Lustre: server umount lustre-MDT0000 complete [ 8747.621301] LustreError: 417136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754666 with bad export cookie 5244167018872018991 [ 8747.623238] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8747.626131] LustreError: 417136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8747.762432] Lustre: server umount lustre-MDT0001 complete [ 8752.025046] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8753.229235] Key type lgssc unregistered [ 8753.387602] LNet: 420693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8753.391416] LNetError: 420693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8753.399472] LNet: Removed LNI 192.168.206.151@tcp [ 8753.785616] Key type .llcrypt unregistered [ 8753.787362] Key type ._llcrypt unregistered [ 8765.634572] Key type ._llcrypt registered [ 8765.636729] Key type .llcrypt registered [ 8765.696622] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8766.110616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8766.121964] alg: No test for adler32 (adler32-zlib) [ 8767.027693] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 8767.145823] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8768.751253] Key type lgssc registered [ 8769.228878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8772.316104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8772.322581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8773.522791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8775.140251] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8778.748912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8778.974319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8780.819716] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8784.169000] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8786.748508] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8789.849318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8790.035898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8793.056509] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8796.402831] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8798.250416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8804.834395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8804.835514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8804.835671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8804.840658] Lustre: Skipped 2 previous similar messages [ 8808.144895] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 8808.148657] Lustre: Skipped 2 previous similar messages [ 8809.957449] LustreError: 422097:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8809.962682] LustreError: 422097:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 8809.979772] Lustre: server umount lustre-MDT0000 complete [ 8811.025881] LustreError: 422078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.51@tcp arrived at 1788754730 with bad export cookie 1210832179987728677 [ 8811.033993] LustreError: 422078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8811.917976] LustreError: 422079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788754731 with bad export cookie 1210832179987728061 [ 8811.927550] LustreError: 422079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8811.927976] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8815.072580] LustreError: 422098:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8815.072898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8815.079894] LustreError: 422098:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 8815.089077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8816.096149] LustreError: 422836:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8816.103490] LustreError: 422836:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 8818.077568] Lustre: server umount lustre-OST0000 complete [ 8821.175630] LustreError: 422826:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000b75bf9b x1875645233654272/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8821.243077] Lustre: server umount lustre-MDT0001 complete [ 8825.549222] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8826.760700] Key type lgssc unregistered [ 8826.926790] LNet: 425779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8826.931571] LNetError: 425779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8826.940574] LNet: Removed LNI 192.168.206.151@tcp [ 8827.279921] Key type .llcrypt unregistered [ 8827.281096] Key type ._llcrypt unregistered [ 8838.196625] Key type ._llcrypt registered [ 8838.198339] Key type .llcrypt registered [ 8838.242785] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8838.734960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8838.783848] alg: No test for adler32 (adler32-zlib) [ 8839.670795] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 8839.778732] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8841.399259] Key type lgssc registered [ 8841.919337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8845.597873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8845.608571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8846.822732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8848.665381] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8852.568399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8852.754791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8854.505470] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8857.508741] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8859.828471] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8862.737239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8862.889164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8865.188944] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8868.283175] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8871.082780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8877.536385] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8877.540473] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.547572] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8880.841379] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 8881.636782] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8882.845349] Lustre: server umount lustre-MDT0001 complete [ 8886.753162] LustreError: 428407:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8886.753717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8886.754886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.755642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8886.755649] Lustre: Skipped 2 previous similar messages [ 8886.759819] LustreError: 428407:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 8891.028187] Lustre: server umount lustre-OST0000 complete [ 8891.873620] LustreError: 427167:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8892.687396] Lustre: server umount lustre-MDT0000 complete [ 8898.268938] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8899.476861] Key type lgssc unregistered [ 8899.626489] LNet: 430847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8899.629664] LNetError: 430847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8899.637406] LNet: Removed LNI 192.168.206.151@tcp [ 8899.996221] Key type .llcrypt unregistered [ 8899.997976] Key type ._llcrypt unregistered [ 8910.774186] Key type ._llcrypt registered [ 8910.776148] Key type .llcrypt registered [ 8910.827907] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8918.581238] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8919.094031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8919.111944] alg: No test for adler32 (adler32-zlib) [ 8920.013644] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 8920.136697] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8921.735398] Key type lgssc registered [ 8922.249182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8940.674768] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8947.208896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8947.216300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8948.314882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8948.331437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8948.370782] Lustre: lustre-MDT0000: new disk, initializing [ 8948.399943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8948.407014] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8950.031958] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8957.482082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8957.631994] Lustre: lustre-OST0000: new disk, initializing [ 8957.634795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8957.638277] Lustre: Skipped 1 previous similar message [ 8957.641823] Lustre: 435849:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8957.685536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8957.749268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8957.753367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8957.774120] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8960.377474] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8965.090456] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8972.407153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8972.479577] Lustre: 434831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8972.514314] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8972.563238] Lustre: lustre-MDT0001: new disk, initializing [ 8972.617774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8972.642049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8972.648238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8974.731626] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8979.666447] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8979.785610] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8981.949955] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8983.085989] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 8987.108671] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8987.110418] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8987.113903] Lustre: Skipped 1 previous similar message [ 8987.114243] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8987.136844] Lustre: Skipped 1 previous similar message [ 8988.888486] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 8991.411714] Lustre: server umount lustre-MDT0001 complete [ 8997.344882] LustreError: 434838:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8997.357351] LustreError: 434838:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 4 previous similar messages [ 9002.467110] LustreError: 434836:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9002.475203] LustreError: 434836:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9007.585200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9007.586420] LustreError: 438158:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9007.591698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9010.774230] Lustre: server umount lustre-OST0000 complete [ 9018.436420] Lustre: server umount lustre-MDT0000 complete [ 9023.942260] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9025.157048] Key type lgssc unregistered [ 9025.308579] LNet: 439599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9025.311868] LNetError: 439599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9025.320661] LNet: Removed LNI 192.168.206.151@tcp [ 9025.667882] Key type .llcrypt unregistered [ 9025.670569] Key type ._llcrypt unregistered [ 9042.378106] Key type ._llcrypt registered [ 9042.379504] Key type .llcrypt registered [ 9042.426230] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9042.556576] Key type .llcrypt unregistered [ 9042.558064] Key type ._llcrypt unregistered [ 9049.820767] Key type ._llcrypt registered [ 9049.822211] Key type .llcrypt registered [ 9049.867366] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9050.311870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9050.326605] alg: No test for adler32 (adler32-zlib) [ 9051.235472] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 9051.342287] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9052.935290] Key type lgssc registered [ 9053.384926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9064.430609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9064.439624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9065.549598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9065.567934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9065.616918] Lustre: lustre-MDT0000: new disk, initializing [ 9065.666756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9065.676048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9067.751280] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9073.497798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9073.572866] Lustre: 442938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9073.604455] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9073.607372] Lustre: Skipped 1 previous similar message [ 9073.650444] Lustre: lustre-MDT0001: new disk, initializing [ 9073.700788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9073.730120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9073.737823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9075.755847] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9080.511827] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9082.782823] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9085.580249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9085.695990] Lustre: lustre-OST0000: new disk, initializing [ 9085.700345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9085.704222] Lustre: 444907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9085.742969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9088.344984] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9089.215993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9089.222316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9089.258823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9092.939412] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9099.231945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9099.235140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9099.240600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9099.747637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9099.749821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9099.753175] Lustre: Skipped 1 previous similar message [ 9099.757132] Lustre: Skipped 1 previous similar message [ 9102.024645] Lustre: server umount lustre-MDT0000 complete [ 9103.719604] LustreError: 442930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755022 with bad export cookie 5927958714438675468 [ 9103.724753] LustreError: 442930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9103.725317] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9103.869345] Lustre: server umount lustre-MDT0001 complete [ 9115.773091] Lustre: server umount lustre-OST0000 complete [ 9120.493794] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:24:00 (1788755040) [ 9124.069843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9124.350352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9126.246919] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9129.440605] LustreError: 446932:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9129.447925] LustreError: 446932:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [ 9129.837420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9129.984411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9131.600399] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9134.780711] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9137.152117] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9153.189045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9153.272405] Lustre: Found index 0 for lustre-OST0000, updating log [ 9153.346773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9155.659798] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9158.584970] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9170.789948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 9176.032783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9176.036061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9176.036942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9176.040221] Lustre: Skipped 1 previous similar message [ 9177.680337] Lustre: server umount lustre-OST0000 complete [ 9190.313570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9190.314602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9190.318686] Lustre: Skipped 1 previous similar message [ 9190.322461] Lustre: Skipped 4 previous similar messages [ 9195.488467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9195.491676] Lustre: Skipped 1 previous similar message [ 9196.437136] Lustre: server umount lustre-MDT0000 complete [ 9197.773086] LustreError: 446912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755116 with bad export cookie 5927958714438676861 [ 9197.776143] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9197.777380] LustreError: 446912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9197.898124] Lustre: server umount lustre-MDT0001 complete [ 9201.222121] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:25:21 (1788755121) [ 9212.127632] LNet: 1 local NIs in recovery (showing 1): 192.168.206.151@tcp [ 9785.281279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9785.459517] LustreError: 453353:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9785.490993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9787.203472] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9790.945782] LustreError: 453354:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9791.113728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9791.273131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9792.902234] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9796.010319] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9798.359511] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9814.179862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9814.286297] Lustre: Found index 0 for lustre-OST0000, updating log [ 9814.413968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9816.676251] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9819.371873] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9823.712877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9823.719878] LustreError: Skipped 1 previous similar message [ 9823.722087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9823.728149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9826.279310] Lustre: server umount lustre-OST0000 complete [ 9828.896869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9828.898273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9828.900976] Lustre: Skipped 1 previous similar message [ 9830.944364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9830.947123] Lustre: Skipped 4 previous similar messages [ 9835.039975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9835.042120] Lustre: Skipped 7 previous similar messages [ 9840.863261] LNet: 1 local NIs in recovery (showing 1): 192.168.206.151@tcp [ 9842.143130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9842.206490] Lustre: server umount lustre-MDT0000 complete [ 9842.210938] LustreError: 455692:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9842.217209] LustreError: 455692:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [ 9843.525199] LustreError: 453335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755762 with bad export cookie 5927958714438677932 [ 9843.528560] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9843.529338] LustreError: 453335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9843.644173] Lustre: server umount lustre-MDT0001 complete [ 9861.709635] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 9865.462326] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9878.974454] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 00:36:38 (1788755798) [ 9885.326402] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9886.489035] Key type lgssc unregistered [ 9890.719344] LNet: 459980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9890.723092] LNetError: 459980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9947.045282] LNet: Removed LNI 192.168.206.151@tcp [ 9947.411149] Key type .llcrypt unregistered [ 9947.412806] Key type ._llcrypt unregistered [ 9955.342691] Key type ._llcrypt registered [ 9955.344368] Key type .llcrypt registered [ 9955.392522] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9955.748074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9955.812075] alg: No test for adler32 (adler32-zlib) [ 9956.699757] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [ 9956.802533] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9958.392323] Key type lgssc registered [ 9958.817074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9960.524250] LNet: Added LNI 192.168.206.151@tcp100 [8/256/0/180] [ 9960.923668] LNet: Added LNI 192.168.206.151@tcp101 [8/256/0/180] [ 9961.302448] LNet: Added LNI 192.168.206.151@tcp102 [8/256/0/180] [ 9961.758425] LNet: Added LNI 192.168.206.151@tcp103 [8/256/0/180] [ 9962.174280] LNet: Added LNI 192.168.206.151@tcp104 [8/256/0/180] [ 9962.553109] LNet: Added LNI 192.168.206.151@tcp105 [8/256/0/180] [ 9962.931800] LNet: Added LNI 192.168.206.151@tcp106 [8/256/0/180] [ 9963.312977] LNet: Added LNI 192.168.206.151@tcp107 [8/256/0/180] [ 9963.673792] LNet: Added LNI 192.168.206.151@tcp108 [8/256/0/180] [ 9964.058364] LNet: Added LNI 192.168.206.151@tcp109 [8/256/0/180] [ 9964.439577] LNet: Added LNI 192.168.206.151@tcp110 [8/256/0/180] [ 9964.857770] LNet: Added LNI 192.168.206.151@tcp111 [8/256/0/180] [ 9965.226228] LNet: Added LNI 192.168.206.151@tcp112 [8/256/0/180] [ 9965.592724] LNet: Added LNI 192.168.206.151@tcp113 [8/256/0/180] [ 9965.954619] LNet: Added LNI 192.168.206.151@tcp114 [8/256/0/180] [ 9966.311406] LNet: Added LNI 192.168.206.151@tcp115 [8/256/0/180] [ 9966.688512] LNet: Added LNI 192.168.206.151@tcp116 [8/256/0/180] [ 9967.053633] LNet: Added LNI 192.168.206.151@tcp117 [8/256/0/180] [ 9967.412688] LNet: Added LNI 192.168.206.151@tcp118 [8/256/0/180] [ 9967.770053] LNet: Added LNI 192.168.206.151@tcp119 [8/256/0/180] [ 9968.115578] LNet: Added LNI 192.168.206.151@tcp120 [8/256/0/180] [ 9968.455909] LNet: Added LNI 192.168.206.151@tcp121 [8/256/0/180] [ 9968.787516] LNet: Added LNI 192.168.206.151@tcp122 [8/256/0/180] [ 9969.133507] LNet: Added LNI 192.168.206.151@tcp123 [8/256/0/180] [ 9969.466565] LNet: Added LNI 192.168.206.151@tcp124 [8/256/0/180] [ 9969.796682] LNet: Added LNI 192.168.206.151@tcp125 [8/256/0/180] [ 9970.172611] LNet: Added LNI 192.168.206.151@tcp126 [8/256/0/180] [ 9970.520699] LNet: Added LNI 192.168.206.151@tcp127 [8/256/0/180] [ 9970.875765] LNet: Added LNI 192.168.206.151@tcp128 [8/256/0/180] [ 9971.209108] LNet: Added LNI 192.168.206.151@tcp129 [8/256/0/180] [ 9971.544883] LNet: Added LNI 192.168.206.151@tcp130 [8/256/0/180] [ 9971.886343] LNet: Added LNI 192.168.206.151@tcp131 [8/256/0/180] [ 9972.226487] LNet: Added LNI 192.168.206.151@tcp132 [8/256/0/180] [ 9972.562414] LNet: Added LNI 192.168.206.151@tcp133 [8/256/0/180] [ 9972.879461] LNet: Added LNI 192.168.206.151@tcp134 [8/256/0/180] [ 9973.227091] LNet: Added LNI 192.168.206.151@tcp135 [8/256/0/180] [ 9975.752926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9975.759747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9976.817214] Lustre: 462972:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.151@tcp: can't add NID 192.168.206.151@tcp130: rc = -75 [ 9976.844807] LNetError: 460437:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [ 9976.848833] LNetError: Protocol error connecting to 192.168.206.151@tcp116 at host 192.168.206.151:988: is it running a compatible version of Lustre? [ 9976.852907] LNetError: Skipped 3 previous similar messages [ 9976.905255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9976.920527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9976.959864] Lustre: lustre-MDT0000: new disk, initializing [ 9976.988235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9976.995565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9977.311418] LNet: 8 local NIs in recovery (showing 8): 192.168.206.151@tcp118, 192.168.206.151@tcp121, 192.168.206.151@tcp122, 192.168.206.151@tcp124, 192.168.206.151@tcp127, 192.168.206.151@tcp125, 192.168.206.151@tcp126, 192.168.206.151@tcp129 [ 9978.336096] LNetError: 462997:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [ 9978.340714] LNetError: 462997:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [ 9978.344801] LNetError: Protocol error connecting to 192.168.206.151@tcp118 at host 192.168.206.151:988: is it running a compatible version of Lustre? [ 9978.350557] LNetError: Skipped 14 previous similar messages [ 9978.607328] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9979.364153] LNetError: 460437:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [ 9979.369643] LNetError: Protocol error connecting to 192.168.206.151@tcp125 at host 192.168.206.151:988: is it running a compatible version of Lustre? [ 9979.374105] LNetError: Skipped 1 previous similar message [ 9983.200770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9983.260898] Lustre: 462987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9983.294391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9983.296898] Lustre: Skipped 1 previous similar message [ 9983.336531] Lustre: lustre-MDT0001: new disk, initializing [ 9983.351076] Lustre: 463789:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.151@tcp: can't add NID 192.168.206.151@tcp130: rc = -75 [ 9983.354780] Lustre: 463789:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9983.404126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9983.420084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9983.425142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9983.457396] LNetError: 462997:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [ 9983.462194] LNetError: 462997:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [ 9983.465863] LNetError: Protocol error connecting to 192.168.206.151@tcp125 at host 192.168.206.151:988: is it running a compatible version of Lustre? [ 9985.050927] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9988.970188] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9990.980479] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9991.648273] LNetError: 460436:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [ 9991.652249] LNetError: Protocol error connecting to 192.168.206.151@tcp125 at host 192.168.206.151:988: is it running a compatible version of Lustre? [ 9993.266249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9993.376241] Lustre: lustre-OST0000: new disk, initializing [ 9993.377970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9993.380823] Lustre: 464963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9993.415336] Lustre: 464968:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.151@tcp: can't add NID 192.168.206.151@tcp130: rc = -75 [ 9993.418857] Lustre: 464968:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [ 9993.438709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9995.466610] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9999.408426] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10002.064557] Lustre: server umount lustre-OST0000 complete [10002.421183] Lustre: 465958:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.151@tcp: can't add NID 192.168.206.151@tcp130: rc = -75 [10002.424950] Lustre: 465958:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [10003.936334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10003.939978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10003.945426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10007.009037] LNetError: 462993:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.151 [10007.013302] LNetError: Protocol error connecting to 192.168.206.151@tcp125 at host 192.168.206.151:988: is it running a compatible version of Lustre? [10007.520166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10007.520724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10009.594300] Lustre: server umount lustre-MDT0000 complete [10011.264461] LustreError: 462982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788755930 with bad export cookie 6008745597090804684 [10011.266690] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10011.269790] LustreError: 462982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10011.418092] Lustre: server umount lustre-MDT0001 complete [10016.169042] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [10017.530479] Key type lgssc unregistered [10017.749791] LNet: 466867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10017.753326] LNetError: 466867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10017.756355] LNet: Removed LNI 192.168.206.151@tcp [10018.783414] LNet: Removed LNI 192.168.206.151@tcp100 [10018.787461] LNetError: 466867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10018.791894] LNetError: 466867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10018.795913] LNet: Removed LNI 192.168.206.151@tcp101 [10019.807136] LNet: Removed LNI 192.168.206.151@tcp102 [10019.808601] LNetError: 466867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10019.811851] LNetError: 466867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10019.815262] LNet: Removed LNI 192.168.206.151@tcp103 [10020.831502] LNet: Removed LNI 192.168.206.151@tcp104 [10021.856173] LNet: Removed LNI 192.168.206.151@tcp105 [10021.859984] LNetError: 466867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10021.865579] LNetError: 466867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [10021.871440] LNet: Removed LNI 192.168.206.151@tcp106 [10022.880352] LNet: Removed LNI 192.168.206.151@tcp107 [10023.903191] LNet: Removed LNI 192.168.206.151@tcp108 [10024.927156] LNet: Removed LNI 192.168.206.151@tcp109 [10025.951145] LNet: Removed LNI 192.168.206.151@tcp110 [10025.952942] LNetError: 466867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10025.957904] LNetError: 466867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [10025.962820] LNet: Removed LNI 192.168.206.151@tcp111 [10026.975909] LNet: Removed LNI 192.168.206.151@tcp112 [10027.999993] LNet: Removed LNI 192.168.206.151@tcp113 [10029.023322] LNet: Removed LNI 192.168.206.151@tcp114 [10030.047147] LNet: Removed LNI 192.168.206.151@tcp115 [10031.071142] LNet: Removed LNI 192.168.206.151@tcp116 [10032.095167] LNet: Removed LNI 192.168.206.151@tcp117 [10033.119159] LNet: Removed LNI 192.168.206.151@tcp118 [10034.143139] LNet: Removed LNI 192.168.206.151@tcp119 [10034.145650] LNetError: 466867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10034.148306] LNetError: 466867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [10034.151769] LNet: Removed LNI 192.168.206.151@tcp120 [10035.167193] LNet: Removed LNI 192.168.206.151@tcp121 [10036.192331] LNet: Removed LNI 192.168.206.151@tcp122 [10037.215271] LNet: Removed LNI 192.168.206.151@tcp123 [10038.239236] LNet: Removed LNI 192.168.206.151@tcp124 [10038.241571] LNet: Removed LNI 192.168.206.151@tcp125 [10039.263305] LNet: Removed LNI 192.168.206.151@tcp126 [10040.287181] LNet: Removed LNI 192.168.206.151@tcp127 [10041.311120] LNet: Removed LNI 192.168.206.151@tcp128 [10042.335143] LNet: Removed LNI 192.168.206.151@tcp129 [10042.337733] LNet: Removed LNI 192.168.206.151@tcp130 [10042.339589] LNet: Removed LNI 192.168.206.151@tcp131 [10042.341951] LNet: Removed LNI 192.168.206.151@tcp132 [10042.343652] LNet: Removed LNI 192.168.206.151@tcp133 [10042.347562] LNet: Removed LNI 192.168.206.151@tcp134 [10042.355995] LNet: Removed LNI 192.168.206.151@tcp135 [10042.741551] Key type .llcrypt unregistered [10042.742828] Key type ._llcrypt unregistered [10052.639460] Key type ._llcrypt registered [10052.641021] Key type .llcrypt registered [10052.698688] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [10059.699519] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10060.164324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10060.208788] alg: No test for adler32 (adler32-zlib) [10061.105423] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [10061.219464] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [10062.831352] Key type lgssc registered [10063.316653] Lustre: Echo OBD driver; http://www.lustre.org/ [10077.917715] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 00:39:57 (1788755997) [10085.938919] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [10087.193820] Key type lgssc unregistered [10087.339533] LNet: 470924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10087.342966] LNetError: 470924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10087.356683] LNet: Removed LNI 192.168.206.151@tcp [10087.718046] Key type .llcrypt unregistered [10087.720425] Key type ._llcrypt unregistered [10095.852301] Key type ._llcrypt registered [10095.853673] Key type .llcrypt registered [10095.901467] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10096.409666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10096.468555] alg: No test for adler32 (adler32-zlib) [10097.382221] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [10097.512476] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [10099.127368] Key type lgssc registered [10099.616618] Lustre: Echo OBD driver; http://www.lustre.org/ [10104.006128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10104.012960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10105.130868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10105.149406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10105.191587] Lustre: lustre-MDT0000: new disk, initializing [10105.233753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10105.243379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10107.109358] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10112.224340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10112.280132] Lustre: 472110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10112.320655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10112.323470] Lustre: Skipped 1 previous similar message [10112.363665] Lustre: lustre-MDT0001: new disk, initializing [10112.405549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10112.422139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10112.426977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10114.222074] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10118.816843] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10121.146676] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10124.149221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10124.272186] Lustre: lustre-OST0000: new disk, initializing [10124.274576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10124.278512] Lustre: 474080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10124.321293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10126.825854] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10131.498249] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10133.005141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10133.011357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10133.071468] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10133.208646] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing lnet_if_list [10135.576221] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [10135.596033] LNet: Added LNI 192.168.206.151@tcp10 [8/256/0/180] [10138.063106] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [10138.079220] LNet: Added LNI 192.168.206.151@tcp11 [8/256/0/180] [10139.793793] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [10139.809717] LNet: Added LNI 192.168.206.151@tcp12 [8/256/0/180] [10141.661190] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [10141.676554] LNet: Added LNI 192.168.206.151@tcp13 [8/256/0/180] [10143.460050] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [10143.475115] LNet: Added LNI 192.168.206.151@tcp14 [8/256/0/180] [10145.412947] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [10145.431251] LNet: Added LNI 192.168.206.151@tcp15 [8/256/0/180] [10147.274079] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [10147.290529] LNet: Added LNI 192.168.206.151@tcp16 [8/256/0/180] [10149.155887] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10149.172986] LNet: Added LNI 192.168.206.151@tcp17 [8/256/0/180] [10151.110172] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [10151.124840] LNet: Added LNI 192.168.206.151@tcp18 [8/256/0/180] [10153.328343] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [10153.345178] LNet: Added LNI 192.168.206.151@tcp19 [8/256/0/180] [10155.485634] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [10155.505103] LNet: Added LNI 192.168.206.151@tcp20 [8/256/0/180] [10157.708618] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [10157.726551] LNet: Added LNI 192.168.206.151@tcp21 [8/256/0/180] [10160.009643] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [10160.025702] LNet: Added LNI 192.168.206.151@tcp22 [8/256/0/180] [10162.328269] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [10162.344273] LNet: Added LNI 192.168.206.151@tcp23 [8/256/0/180] [10164.841294] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [10164.856438] LNet: Added LNI 192.168.206.151@tcp24 [8/256/0/180] [10167.225117] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [10167.241156] LNet: Added LNI 192.168.206.151@tcp25 [8/256/0/180] [10169.614497] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [10169.631508] LNet: Added LNI 192.168.206.151@tcp26 [8/256/0/180] [10172.144668] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [10172.165211] LNet: Added LNI 192.168.206.151@tcp27 [8/256/0/180] [10175.002584] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [10175.017876] LNet: Added LNI 192.168.206.151@tcp28 [8/256/0/180] [10177.765424] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [10177.783379] LNet: Added LNI 192.168.206.151@tcp29 [8/256/0/180] [10180.845811] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [10180.862604] LNet: Added LNI 192.168.206.151@tcp30 [8/256/0/180] [10183.765613] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [10183.782155] LNet: Added LNI 192.168.206.151@tcp31 [8/256/0/180] [10186.580792] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [10186.595880] LNet: Added LNI 192.168.206.151@tcp32 [8/256/0/180] [10189.455570] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [10189.472204] LNet: Added LNI 192.168.206.151@tcp33 [8/256/0/180] [10192.250385] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10192.267119] LNet: Added LNI 192.168.206.151@tcp34 [8/256/0/180] [10195.326841] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10195.342475] LNet: Added LNI 192.168.206.151@tcp35 [8/256/0/180] [10198.379505] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10198.396102] LNet: Added LNI 192.168.206.151@tcp36 [8/256/0/180] [10201.251668] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10201.265640] LNet: Added LNI 192.168.206.151@tcp37 [8/256/0/180] [10204.303206] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10204.320757] LNet: Added LNI 192.168.206.151@tcp38 [8/256/0/180] [10207.249208] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10207.266116] LNet: Added LNI 192.168.206.151@tcp39 [8/256/0/180] [10210.347204] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10210.366284] LNet: Added LNI 192.168.206.151@tcp40 [8/256/0/180] [10213.611023] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10213.625966] LNet: Added LNI 192.168.206.151@tcp41 [8/256/0/180] [10216.545521] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10216.560886] LNet: Added LNI 192.168.206.151@tcp42 [8/256/0/180] [10219.623968] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10219.637914] LNet: Added LNI 192.168.206.151@tcp43 [8/256/0/180] [10222.984205] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10223.000409] LNet: Added LNI 192.168.206.151@tcp44 [8/256/0/180] [10226.121344] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10226.137505] LNet: Added LNI 192.168.206.151@tcp45 [8/256/0/180] [10229.324919] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10229.340880] LNet: Added LNI 192.168.206.151@tcp46 [8/256/0/180] [10232.467226] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10232.484429] LNet: Added LNI 192.168.206.151@tcp47 [8/256/0/180] [10235.870160] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10235.885823] LNet: Added LNI 192.168.206.151@tcp48 [8/256/0/180] [10239.223059] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10239.236764] LNet: Added LNI 192.168.206.151@tcp49 [8/256/0/180] [10242.711132] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10242.727165] LNet: Added LNI 192.168.206.151@tcp50 [8/256/0/180] [10255.844694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10255.844701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10255.846134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10255.849997] Lustre: Skipped 2 previous similar messages [10255.853236] Lustre: Skipped 1 previous similar message [10257.545992] Lustre: server umount lustre-MDT0000 complete [10259.317039] LustreError: 472102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756178 with bad export cookie 2285031287090530222 [10259.320720] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10259.326032] LustreError: 472102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10259.475535] Lustre: server umount lustre-MDT0001 complete [10268.127382] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10271.827346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10277.279506] Lustre: 471382:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788756180/real 1788756180] req@00000000c82ac9e6 x1875646628328960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788756196 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10277.287942] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10281.375167] Lustre: 471384:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788756184/real 1788756184] req@00000000646c6eea x1875646628329088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788756200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10282.399151] Lustre: 471384:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788756185/real 1788756185] req@0000000076b431f0 x1875646628329216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788756201 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10284.447445] LustreError: 471380:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d817072e x1875646628330112/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10284.587350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10285.599124] Lustre: 471382:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788756188/real 1788756188] req@00000000c16736ec x1875646628329472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788756204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10286.242239] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10289.349995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10289.507552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10290.528109] Lustre: 471381:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788756193/real 1788756193] req@0000000036ab4d8b x1875646628329600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788756209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10291.072874] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10293.631134] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10294.758168] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10295.712330] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10296.804212] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10304.995458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10304.999941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10305.006486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10307.040917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10307.044853] Lustre: Skipped 2 previous similar messages [10310.168828] Lustre: server umount lustre-OST0000 complete [10312.160793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10312.161531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10312.166727] Lustre: Skipped 2 previous similar messages [10317.280974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10317.284654] Lustre: Skipped 2 previous similar messages [10317.686202] Lustre: server umount lustre-MDT0000 complete [10319.286422] LustreError: 474079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756238 with bad export cookie 2285031287090532392 [10319.290089] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10319.292907] LustreError: 474079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10319.423399] Lustre: server umount lustre-MDT0001 complete [10323.975294] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [10325.158054] Key type lgssc unregistered [10325.344814] LNet: 501041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10325.350977] LNetError: 501041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10325.354341] LNet: Removed LNI 192.168.206.151@tcp [10325.356197] LNet: Removed LNI 192.168.206.151@tcp10 [10325.358253] LNet: Removed LNI 192.168.206.151@tcp11 [10325.359736] LNet: Removed LNI 192.168.206.151@tcp12 [10325.363148] LNet: Removed LNI 192.168.206.151@tcp13 [10325.365709] LNet: Removed LNI 192.168.206.151@tcp14 [10325.367889] LNet: Removed LNI 192.168.206.151@tcp15 [10325.370321] LNet: Removed LNI 192.168.206.151@tcp16 [10325.372176] LNet: Removed LNI 192.168.206.151@tcp17 [10325.373893] LNet: Removed LNI 192.168.206.151@tcp18 [10325.375694] LNet: Removed LNI 192.168.206.151@tcp19 [10325.378418] LNet: Removed LNI 192.168.206.151@tcp20 [10325.380764] LNet: Removed LNI 192.168.206.151@tcp21 [10325.382845] LNet: Removed LNI 192.168.206.151@tcp22 [10325.385157] LNet: Removed LNI 192.168.206.151@tcp23 [10325.386534] LNet: Removed LNI 192.168.206.151@tcp24 [10325.388301] LNet: Removed LNI 192.168.206.151@tcp25 [10325.390210] LNet: Removed LNI 192.168.206.151@tcp26 [10325.392333] LNet: Removed LNI 192.168.206.151@tcp27 [10325.394410] LNet: Removed LNI 192.168.206.151@tcp28 [10325.395968] LNet: Removed LNI 192.168.206.151@tcp29 [10325.397990] LNet: Removed LNI 192.168.206.151@tcp30 [10325.399956] LNet: Removed LNI 192.168.206.151@tcp31 [10325.401911] LNet: Removed LNI 192.168.206.151@tcp32 [10325.404135] LNet: Removed LNI 192.168.206.151@tcp33 [10325.406468] LNet: Removed LNI 192.168.206.151@tcp34 [10325.408555] LNet: Removed LNI 192.168.206.151@tcp35 [10325.411322] LNet: Removed LNI 192.168.206.151@tcp36 [10325.413274] LNet: Removed LNI 192.168.206.151@tcp37 [10325.415952] LNet: Removed LNI 192.168.206.151@tcp38 [10325.418323] LNet: Removed LNI 192.168.206.151@tcp39 [10325.420176] LNet: Removed LNI 192.168.206.151@tcp40 [10325.421881] LNet: Removed LNI 192.168.206.151@tcp41 [10325.423661] LNet: Removed LNI 192.168.206.151@tcp42 [10325.425597] LNet: Removed LNI 192.168.206.151@tcp43 [10325.427628] LNet: Removed LNI 192.168.206.151@tcp44 [10325.429594] LNet: Removed LNI 192.168.206.151@tcp45 [10325.431512] LNet: Removed LNI 192.168.206.151@tcp46 [10325.433712] LNet: Removed LNI 192.168.206.151@tcp47 [10325.436076] LNet: Removed LNI 192.168.206.151@tcp48 [10325.437781] LNet: Removed LNI 192.168.206.151@tcp49 [10325.446769] LNet: Removed LNI 192.168.206.151@tcp50 [10325.771487] Key type .llcrypt unregistered [10325.773259] Key type ._llcrypt unregistered [10333.710875] Key type ._llcrypt registered [10333.712373] Key type .llcrypt registered [10333.763378] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [10341.438582] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10341.807863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10341.854024] alg: No test for adler32 (adler32-zlib) [10342.766489] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [10342.884973] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [10344.503198] Key type lgssc registered [10345.009928] Lustre: Echo OBD driver; http://www.lustre.org/ [10361.573388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10361.590393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10362.727237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10362.751176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10362.807156] Lustre: lustre-MDT0000: new disk, initializing [10362.869430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10362.882418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10365.224208] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10371.871735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10371.954687] Lustre: 504210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10371.994065] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10371.996499] Lustre: Skipped 1 previous similar message [10372.041835] Lustre: lustre-MDT0001: new disk, initializing [10372.085844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10372.114670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10372.124835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10374.589403] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10379.293420] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10381.845517] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10385.063057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10385.146253] Lustre: *** cfs_fail_loc=90f, val=0*** [10385.183438] Lustre: lustre-OST0000: new disk, initializing [10385.185578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10385.188691] Lustre: 506222:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10385.226219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10387.600707] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10388.492373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10388.500155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10388.543182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10392.113668] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10398.690913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10398.697584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10398.700097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10398.703746] Lustre: Skipped 1 previous similar message [10401.373850] Lustre: server umount lustre-OST0000 complete [10403.810518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10403.823427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10403.836619] Lustre: Skipped 2 previous similar messages [10408.928265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10408.931619] Lustre: Skipped 1 previous similar message [10409.323714] Lustre: server umount lustre-MDT0000 complete [10411.117539] LustreError: 504200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756330 with bad export cookie 10521177543711958891 [10411.120667] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10411.123654] LustreError: 504200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10411.299547] Lustre: server umount lustre-MDT0001 complete [10416.570529] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [10417.876228] Key type lgssc unregistered [10418.040798] LNet: 508126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10418.043954] LNetError: 508126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10418.057969] LNet: Removed LNI 192.168.206.151@tcp [10418.433325] Key type .llcrypt unregistered [10418.435084] Key type ._llcrypt unregistered [10427.798242] Key type ._llcrypt registered [10427.799508] Key type .llcrypt registered [10427.858981] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10428.288221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10428.318617] alg: No test for adler32 (adler32-zlib) [10429.220471] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [10429.346823] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [10430.951222] Key type lgssc registered [10431.408080] Lustre: Echo OBD driver; http://www.lustre.org/ [10434.919196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10434.927168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10436.174360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10438.136551] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10441.742208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10441.905965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10443.560441] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10446.418364] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10448.545393] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10451.333276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10451.481441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10453.991159] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10457.215536] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10472.404590] loop0: detected capacity change from 0 to 204800000 [10472.488497] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10472.493791] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10472.504395] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10475.186657] loop0: detected capacity change from 0 to 204800000 [10475.259880] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10475.267411] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10475.273441] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10477.613189] loop0: detected capacity change from 0 to 204800000 [10479.866272] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10479.947431] Lustre: fs15246-OST0000: new disk, initializing [10479.950215] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10479.954785] Lustre: 512978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10479.981378] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10482.294730] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10487.012840] loop1: detected capacity change from 0 to 204800000 [10489.640756] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10489.690840] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10489.721559] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10489.762258] Lustre: fs15246-MDT0000: new disk, initializing [10489.824641] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10489.837619] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10489.908140] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10490.975748] Lustre: 514102:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004f359ddf x1875646976393344/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788756426 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10491.591779] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10496.032852] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10496.036573] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10496.044070] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10505.191977] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10505.201680] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10506.983050] Lustre: server umount fs15246-MDT0000 complete [10510.312944] LustreError: 509364:0:(ldlm_lib.c:1202:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10510.325370] LustreError: 509364:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 2 previous similar messages [10511.067532] Lustre: server umount fs15246-OST0000 complete [10517.984393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10517.984750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10517.988098] LustreError: Skipped 1 previous similar message [10517.988651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10517.992502] Lustre: Skipped 1 previous similar message [10520.545626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10520.548710] Lustre: Skipped 2 previous similar messages [10522.281794] Lustre: server umount lustre-OST0000 complete [10524.128057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10524.131791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10524.138449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10530.138301] Lustre: server umount lustre-MDT0000 complete [10530.787259] LustreError: 509367:0:(ldlm_lib.c:1202:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.794482] LustreError: 509367:0:(ldlm_lib.c:1202:target_handle_connect()) Skipped 1 previous similar message [10531.772259] LustreError: 514096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756450 with bad export cookie 14470955755548594683 [10531.775036] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10531.777540] LustreError: 514096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10531.966350] Lustre: server umount lustre-MDT0001 complete [10536.978584] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [10538.355667] Key type lgssc unregistered [10538.521577] LNet: 517192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10538.525269] LNetError: 517192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10538.532067] LNet: Removed LNI 192.168.206.151@tcp [10538.921607] Key type .llcrypt unregistered [10538.923702] Key type ._llcrypt unregistered [10548.159085] Key type ._llcrypt registered [10548.160569] Key type .llcrypt registered [10548.212097] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10548.565899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10548.600052] alg: No test for adler32 (adler32-zlib) [10549.494703] Lustre: Lustre: Build Version: 2.17.58_39_g34d8757 [10549.620828] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [10551.216137] Key type lgssc registered [10551.697988] Lustre: Echo OBD driver; http://www.lustre.org/ [10569.839814] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [10574.460497] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10590.470473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10590.487270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10591.600547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10591.614262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10591.662938] Lustre: lustre-MDT0000: new disk, initializing [10591.705072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10591.715714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10593.486737] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10598.726435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10598.782989] Lustre: 521624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10598.814797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10598.817380] Lustre: Skipped 1 previous similar message [10598.860576] Lustre: lustre-MDT0001: new disk, initializing [10598.903903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10598.922715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10598.930461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10600.737147] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10605.375565] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10607.746198] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10610.636754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10610.762418] Lustre: lustre-OST0000: new disk, initializing [10610.765228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10610.770480] Lustre: 523594:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10610.809638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10613.405353] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10616.841192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10616.846661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10616.898655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10618.302279] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10620.307916] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10620.407204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10622.368258] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10622.453879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10627.041556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10627.043921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10627.047267] Lustre: Skipped 1 previous similar message [10629.555558] Lustre: server umount lustre-OST0000 complete [10632.161581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10632.163593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10632.167467] Lustre: Skipped 1 previous similar message [10632.172496] Lustre: Skipped 2 previous similar messages [10637.280160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10637.282480] Lustre: Skipped 1 previous similar message [10637.482403] Lustre: server umount lustre-MDT0000 complete [10639.226286] LustreError: 523593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788756558 with bad export cookie 14864824671430604653 [10639.227899] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10639.233533] LustreError: 523593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10639.397402] Lustre: server umount lustre-MDT0001 complete [10651.216431] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [10656.040913] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10670.055583] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10347 sec ========= 00:49:49 (1788756589) [10670.905371] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:49:50 (1788756590) === [10672.634929] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:49:52 (1788756592) === [10690.259297] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [10691.590229] Key type lgssc unregistered [10691.737443] LNet: 530224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10691.742651] LNetError: 530224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10691.760393] LNet: Removed LNI 192.168.206.151@tcp [10692.194559] Key type .llcrypt unregistered [10692.196161] Key type ._llcrypt unregistered