[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 445182648 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002311] x2apic enabled [ 0.003017] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006511] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008006] pid_max: default: 32768 minimum: 301 [ 0.009116] LSM: Security Framework initializing [ 0.010035] Yama: becoming mindful. [ 0.011030] SELinux: Initializing. [ 0.012066] *** VALIDATE selinux *** [ 0.020715] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024357] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026104] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027096] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028094] *** VALIDATE tmpfs *** [ 0.030303] *** VALIDATE proc *** [ 0.031215] *** VALIDATE cgroup *** [ 0.032006] *** VALIDATE cgroup2 *** [ 0.033273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035128] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039005] Speculative Store Bypass: Vulnerable [ 0.042000] debug: unmapping init [mem 0xffffffff8d859000-0xffffffff8d860fff] [ 0.043186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044785] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045016] ... version: 2 [ 0.046016] ... bit width: 48 [ 0.046793] ... generic registers: 4 [ 0.047007] ... value mask: 0000ffffffffffff [ 0.048008] ... max period: 00007fffffffffff [ 0.049010] ... fixed-purpose events: 3 [ 0.049935] ... event mask: 000000070000000f [ 0.050240] rcu: Hierarchical SRCU implementation. [ 0.052250] smp: Bringing up secondary CPUs ... [ 0.053568] x86: Booting SMP configuration: [ 0.054027] .... node #0, CPUs: #1 #2 #3 [ 0.058068] smp: Brought up 1 node, 4 CPUs [ 0.059808] smpboot: Max logical packages: 1 [ 0.060009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.126978] node 0 deferred pages initialised in 65ms [ 0.130113] devtmpfs: initialized [ 0.131348] x86/mm: Memory block size: 128MB [ 0.137368] gcov: version magic: 0x41383552 [ 0.140297] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146318] pinctrl core: initialized pinctrl subsystem [ 0.149150] [ 0.149752] ************************************************************* [ 0.152012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153010] ** ** [ 0.155009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156009] ** ** [ 0.158012] ** This means that this kernel is built to expose internal ** [ 0.161014] ** IOMMU data structures, which may compromise security on ** [ 0.163053] ** your system. ** [ 0.165011] ** ** [ 0.167008] ** If you see this message and you are not debugging the ** [ 0.169008] ** kernel, report this immediately to your vendor! ** [ 0.170006] ** ** [ 0.171012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173006] ************************************************************* [ 0.175567] NET: Registered protocol family 16 [ 0.177359] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.180111] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.182067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.185204] cpuidle: using governor menu [ 0.186508] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.189695] PCI: Using configuration type 1 for base access [ 0.191155] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.201192] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.204030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.208098] cryptd: max_cpu_qlen set to 1000 [ 0.210742] ACPI: Added _OSI(Module Device) [ 0.214020] ACPI: Added _OSI(Processor Device) [ 0.217016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.218006] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223363] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.229355] ACPI: Interpreter enabled [ 0.230042] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230897] ACPI: Using IOAPIC for interrupt routing [ 0.232059] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.234536] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.268200] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.278141] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.283544] acpiphp: Slot [2] registered [ 0.284099] acpiphp: Slot [5] registered [ 0.285027] acpiphp: Slot [6] registered [ 0.287074] acpiphp: Slot [7] registered [ 0.288067] acpiphp: Slot [8] registered [ 0.289000] acpiphp: Slot [9] registered [ 0.289000] acpiphp: Slot [10] registered [ 0.290069] acpiphp: Slot [3] registered [ 0.292068] acpiphp: Slot [4] registered [ 0.293059] acpiphp: Slot [11] registered [ 0.294060] acpiphp: Slot [12] registered [ 0.295056] acpiphp: Slot [13] registered [ 0.296036] acpiphp: Slot [14] registered [ 0.297068] acpiphp: Slot [15] registered [ 0.298048] acpiphp: Slot [16] registered [ 0.299062] acpiphp: Slot [17] registered [ 0.300000] acpiphp: Slot [18] registered [ 0.300000] acpiphp: Slot [19] registered [ 0.302066] acpiphp: Slot [20] registered [ 0.303086] acpiphp: Slot [21] registered [ 0.305076] acpiphp: Slot [22] registered [ 0.306083] acpiphp: Slot [23] registered [ 0.308075] acpiphp: Slot [24] registered [ 0.309065] acpiphp: Slot [25] registered [ 0.310077] acpiphp: Slot [26] registered [ 0.312086] acpiphp: Slot [27] registered [ 0.313062] acpiphp: Slot [28] registered [ 0.314074] acpiphp: Slot [29] registered [ 0.315054] acpiphp: Slot [30] registered [ 0.316060] acpiphp: Slot [31] registered [ 0.317042] PCI host bridge to bus 0000:00 [ 0.318013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.319012] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.321015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.323013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.324012] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.326016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.331185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.335036] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338326] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347841] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.355059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.365556] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376246] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.380057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.384265] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.412016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.419017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.422727] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.433013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.460015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.469699] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.476018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.486017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.505017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.514256] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.518014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.530018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.543016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.550000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.564016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.572015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.584013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.591755] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.599018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.605018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.626015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.638000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.645017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.651014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.665015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.682948] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.687023] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.689370] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.694796] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.698231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.704256] iommu: Default domain type: Passthrough [ 0.707381] SCSI subsystem initialized [ 0.709121] ACPI: bus type USB registered [ 0.710079] usbcore: registered new interface driver usbfs [ 0.712069] usbcore: registered new interface driver hub [ 0.713048] usbcore: registered new device driver usb [ 0.714131] pps_core: LinuxPPS API ver. 1 registered [ 0.716006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.718091] PTP clock support registered [ 0.719213] EDAC MC: Ver: 3.0.0 [ 0.722120] PCI: Using ACPI for IRQ routing [ 0.723000] NetLabel: Initializing [ 0.724007] NetLabel: domain hash size = 128 [ 0.725006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.727073] NetLabel: unlabeled traffic allowed by default [ 0.728413] vgaarb: loaded [ 0.730622] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.732044] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.739752] clocksource: Switched to clocksource kvm-clock [ 0.854096] VFS: Disk quotas dquot_6.6.0 [ 0.854930] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.856513] *** VALIDATE ramfs *** [ 0.857133] *** VALIDATE hugetlbfs *** [ 0.858703] pnp: PnP ACPI init [ 0.860310] pnp: PnP ACPI: found 6 devices [ 0.873276] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.875259] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.876635] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.877862] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.880015] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.881763] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.883425] NET: Registered protocol family 2 [ 0.884905] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.889264] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.893000] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.903157] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.907371] TCP: Hash tables configured (established 65536 bind 65536) [ 0.910736] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.912712] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.914535] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.916818] NET: Registered protocol family 1 [ 0.919061] RPC: Registered named UNIX socket transport module. [ 0.920626] RPC: Registered udp transport module. [ 0.921986] RPC: Registered tcp transport module. [ 0.923452] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.926375] NET: Registered protocol family 44 [ 0.932242] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.937409] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.943498] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.949513] PCI: CLS 0 bytes, default 64 [ 0.951103] Unpacking initramfs... [ 2.769745] debug: unmapping init [mem 0xffff9962fcc54000-0xffff9962fffbffff] [ 2.776994] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.779218] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.783825] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.395940] Initialise system trusted keyrings [ 3.398101] Key type blacklist registered [ 3.403040] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.414049] zbud: loaded [ 3.417572] *** VALIDATE nfs *** [ 3.419062] *** VALIDATE nfs4 *** [ 3.420911] pstore: using deflate compression [ 3.425460] Platform Keyring initialized [ 3.594416] NET: Registered protocol family 38 [ 3.604786] Key type asymmetric registered [ 3.606990] Asymmetric key parser 'x509' registered [ 3.609678] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.613846] io scheduler mq-deadline registered [ 3.615777] io scheduler kyber registered [ 3.618395] io scheduler bfq registered [ 3.620730] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.624831] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.630400] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.634697] ACPI: Power Button [PWRF] [ 3.643497] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.653427] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.670427] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.685079] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.712714] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.742969] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.774631] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.782495] Non-volatile memory driver v1.3 [ 3.785374] Linux agpgart interface v0.103 [ 3.838793] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.846218] vda: detected capacity change from 0 to 74645504 [ 3.867660] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.870873] vdb: detected capacity change from 0 to 1073741824 [ 3.899091] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.902927] vdc: detected capacity change from 0 to 2621440000 [ 3.921445] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.924605] vdd: detected capacity change from 0 to 2621440000 [ 3.939166] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.941264] vde: detected capacity change from 0 to 4294967296 [ 3.979248] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.985043] vdf: detected capacity change from 0 to 4294967296 [ 3.993904] libphy: Fixed MDIO Bus: probed [ 4.008683] usbcore: registered new interface driver usbserial_generic [ 4.010704] usbserial: USB Serial support registered for generic [ 4.012552] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.015969] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.017639] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.019565] mousedev: PS/2 mouse device common for all mice [ 4.022626] rtc_cmos 00:05: RTC can wake from S4 [ 4.024206] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.027497] rtc_cmos 00:05: registered as rtc0 [ 4.028812] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.031977] intel_pstate: CPU model not supported [ 4.034758] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.037145] hid: raw HID events driver (C) Jiri Kosina [ 4.041528] usbcore: registered new interface driver usbhid [ 4.042645] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.046853] usbhid: USB HID core driver [ 4.051369] drop_monitor: Initializing network drop monitor service [ 4.053291] Initializing XFRM netlink socket [ 4.054892] NET: Registered protocol family 10 [ 4.057527] Segment Routing with IPv6 [ 4.058859] NET: Registered protocol family 17 [ 4.060348] mpls_gso: MPLS GSO support [ 4.066566] RAS: Correctable Errors collector initialized. [ 4.068387] AVX version of gcm_enc/dec engaged. [ 4.069679] AES CTR mode by8 optimization enabled [ 4.168242] sched_clock: Marking stable (4168164500, 0)->(4967053039, -798888539) [ 4.175463] registered taskstats version 1 [ 4.177472] Loading compiled-in X.509 certificates [ 4.179872] zswap: loaded using pool lzo/zbud [ 4.210148] Key type big_key registered [ 4.227959] Key type encrypted registered [ 4.230708] ima: No TPM chip found, activating TPM-bypass! [ 4.232978] ima: Allocated hash algorithm: sha1 [ 4.234269] ima: No architecture policies found [ 4.235735] evm: Initialising EVM extended attributes: [ 4.237525] evm: security.selinux [ 4.238605] evm: security.ima [ 4.240178] evm: security.capability [ 4.241571] evm: HMAC attrs: 0x1 [ 4.244715] rtc_cmos 00:05: setting system clock to 2026-07-18 22:14:43 UTC (1784412883) [ 4.252666] debug: unmapping init [mem 0xffffffff8e803000-0xffffffff8e9fffff] [ 4.256140] debug: unmapping init [mem 0xffffffff8d582000-0xffffffff8d858fff] [ 4.267169] Write protecting the kernel read-only data: 28672k [ 4.270980] debug: unmapping init [mem 0xffffffff8bc03000-0xffffffff8bdfffff] [ 4.273721] debug: unmapping init [mem 0xffffffff8c514000-0xffffffff8c5fffff] [ 4.320892] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.328863] systemd[1]: Detected virtualization kvm. [ 4.330287] systemd[1]: Detected architecture x86-64. [ 4.331757] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.364090] systemd[1]: No hostname configured. [ 4.369318] systemd[1]: Set hostname to . [ 4.370910] random: systemd: uninitialized urandom read (16 bytes read) [ 4.372941] systemd[1]: Initializing machine ID from random generator. [ 4.721955] random: systemd: uninitialized urandom read (16 bytes read) [ 4.724729] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.734368] random: systemd: uninitialized urandom read (16 bytes read) [ 4.736951] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.747433] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.935361] device-mapper: uevent: version 1.0.3 [ 5.937520] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System.[ 7.162088] random: fast init done [ 7.701086] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started udev Coldplug all Devices. [ 8.705696] scsi host0: ata_piix [ 8.863486] scsi host1: ata_piix [ 8.959225] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.963751] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 14.764734] random: crng init done [ 14.767684] random: 7 urandom warning(s) missed due to ratelimiting [ 15.821808] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.635390] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.947422] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.362892] SELinux: Disabled at runtime. [ 20.430443] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.441299] systemd[1]: Detected virtualization kvm. [ 20.443500] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.664269] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.666297] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.676611] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.691225] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.698286] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.723982] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.744039] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch.[ 21.998089] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 23.025494] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.577807] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.668244] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.733989] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.801771] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 29.053294] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit)[ 29.686719] NFS: Registering the id_resolver key type [ 29.691751] Key type id_resolver registered [ 29.695212] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg145-server login: [ 62.209186] libcfs: loading out-of-tree module taints kernel. [ 62.230040] Key type ._llcrypt registered [ 62.231299] Key type .llcrypt registered [ 62.269105] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 69.811673] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 70.349655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.356246] alg: No test for adler32 (adler32-zlib) [ 71.344455] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 71.617212] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 73.223226] Key type lgssc registered [ 73.815726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.766193] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 96.196702] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 101.195550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 101.203669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 102.316239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 102.333684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 102.380494] Lustre: lustre-MDT0000: new disk, initializing [ 102.408433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 102.417980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 103.796117] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.259974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 109.298655] Lustre: 6481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 109.311382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 109.313283] Lustre: Skipped 1 previous similar message [ 109.349523] Lustre: lustre-MDT0001: new disk, initializing [ 109.371178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 109.381511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 109.385188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 110.671850] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 112.997609] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 116.529746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 116.627893] Lustre: lustre-OST0000: new disk, initializing [ 116.629879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 116.633307] Lustre: 8413:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 116.657477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 116.862286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 116.865941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 116.892777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 118.668390] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 124.456179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 124.509292] Lustre: lustre-OST0001: new disk, initializing [ 124.511725] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 124.514771] Lustre: 9489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 124.538429] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 124.883230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 124.887160] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 124.902911] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 126.815541] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 132.735395] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 137.523190] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 138.950007] hrtimer: interrupt took 3998040 ns [ 139.754291] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing check_logdir /tmp/testlogs/ [ 141.559845] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing yml_node [ 143.035207] Lustre: DEBUG MARKER: Client: 2.17.54.173 [ 143.924700] Lustre: DEBUG MARKER: MDS: 2.17.54.173 [ 144.819878] Lustre: DEBUG MARKER: OSS: 2.17.54.173 [ 145.332163] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Jul 18 18:17:04 EDT 2026 [ 151.100142] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 151.603461] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 165.856241] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 165.857076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 165.860011] Lustre: Skipped 2 previous similar messages [ 165.862457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 165.863653] Lustre: Skipped 2 previous similar messages [ 169.511281] Lustre: server umount lustre-MDT0000 complete [ 170.951834] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413050 with bad export cookie 17078321728173179067 [ 170.953019] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 170.956066] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 170.975783] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 170.976069] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 170.980344] Lustre: Skipped 2 previous similar messages [ 170.982760] Lustre: Skipped 1 previous similar message [ 176.095585] LustreError: 12424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 176.095670] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 176.102632] LustreError: 12424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 176.104563] Lustre: Skipped 1 previous similar message [ 181.215919] LustreError: 6487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 181.215981] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 181.222252] LustreError: 6487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 181.224343] Lustre: Skipped 1 previous similar message [ 185.311174] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 185.392858] Lustre: server umount lustre-MDT0001 complete [ 201.183104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 201.229225] Lustre: server umount lustre-OST0000 complete [ 217.055755] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 217.113452] Lustre: server umount lustre-OST0001 complete [ 219.413326] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 222.878274] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 235.953679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.052822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 236.068798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 236.117307] Lustre: lustre-MDT0000: new disk, initializing [ 236.148414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.158736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 237.692802] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 242.235831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 242.276905] Lustre: 15030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 242.407560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 243.916782] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 247.799617] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 249.924287] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 252.551657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 252.689411] Lustre: lustre-OST0000: new disk, initializing [ 252.690634] Lustre: Skipped 1 previous similar message [ 252.692354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 252.694412] Lustre: Skipped 2 previous similar messages [ 252.696503] Lustre: 16999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 252.732152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.735553] Lustre: Skipped 1 previous similar message [ 254.594148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 254.598139] Lustre: Skipped 1 previous similar message [ 254.600281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 254.650974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 254.919613] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 259.077142] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 260.734029] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 260.820216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 262.413641] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 262.490519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 265.186109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 265.186314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.187119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 265.188617] LustreError: Skipped 1 previous similar message [ 265.192906] Lustre: Skipped 1 previous similar message [ 269.265355] Lustre: server umount lustre-OST0000 complete [ 273.375783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 273.378698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.383604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 273.385135] Lustre: Skipped 1 previous similar message [ 285.151162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 285.226618] Lustre: server umount lustre-MDT0000 complete [ 285.665355] LustreError: 16037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 285.671377] LustreError: 16037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 286.880400] LustreError: 15023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413166 with bad export cookie 17078321728173180530 [ 286.883842] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 286.884707] LustreError: 15023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 287.036976] Lustre: server umount lustre-MDT0001 complete [ 288.694308] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:19:27 (1784413167) [ 289.696980] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 290.377649] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:19:29 (1784413169) [ 293.268816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 293.473267] LustreError: 19405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.511200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 295.038305] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 298.374698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 300.045891] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 302.998800] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 305.183157] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 308.436792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 311.146336] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 314.315103] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 324.068166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.069783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 324.070083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 324.070090] Lustre: Skipped 4 previous similar messages [ 324.071756] Lustre: Skipped 2 previous similar messages [ 328.474274] Lustre: server umount lustre-OST0000 complete [ 334.306110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 334.306240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.313495] Lustre: Skipped 1 previous similar message [ 335.963036] Lustre: server umount lustre-MDT0000 complete [ 337.307478] LustreError: 22185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413216 with bad export cookie 17078321728173181412 [ 337.310589] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 337.312431] LustreError: 22185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 337.428246] Lustre: server umount lustre-MDT0001 complete [ 341.292174] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 342.574796] Key type lgssc unregistered [ 342.726402] LNet: 22938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.729222] LNetError: 22938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 342.737531] LNet: Removed LNI 192.168.201.145@tcp [ 343.118147] Key type .llcrypt unregistered [ 343.119443] Key type ._llcrypt unregistered [ 351.707891] Key type ._llcrypt registered [ 351.708939] Key type .llcrypt registered [ 351.749529] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 352.167826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 352.220141] alg: No test for adler32 (adler32-zlib) [ 353.072354] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 353.156106] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 354.743150] Key type lgssc registered [ 355.100698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 357.797469] loop: module loaded [ 357.815757] loop0: detected capacity change from 0 to 204800000 [ 357.941163] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 357.945557] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 358.078963] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 360.051889] loop0: detected capacity change from 0 to 204800000 [ 360.108388] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 360.113317] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 360.117551] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 361.874206] loop0: detected capacity change from 0 to 204800000 [ 363.760386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 363.767094] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.849259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 364.863334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 364.903504] Lustre: lustre-MDT0000: new disk, initializing [ 364.926070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.932592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 366.233387] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 369.617539] loop1: detected capacity change from 0 to 204800000 [ 371.455366] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 371.561817] Lustre: lustre-OST07c6: new disk, initializing [ 371.563218] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 371.564680] Lustre: Skipped 1 previous similar message [ 371.566626] Lustre: 26100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 371.591645] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 373.579189] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 374.334194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 374.336514] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 374.346051] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 375.698234] Lustre: 25085:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 375.702941] Lustre: 25085:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 375.705939] Lustre: 25085:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 375.709065] Lustre: 25085:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 375.711478] Lustre: 25085:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 375.714459] Lustre: 25085:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 386.953613] Lustre: 25115:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 386.956673] Lustre: 25115:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 386.960034] Lustre: 25115:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 386.963271] Lustre: 25115:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 386.967133] Lustre: 25115:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 386.970152] Lustre: 25115:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 390.511663] Lustre: 25115:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 390.515560] Lustre: 25115:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 390.517661] Lustre: 25115:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 390.520103] Lustre: 25115:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.523501] Lustre: 25115:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 390.525921] Lustre: 25115:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.528809] Lustre: 25115:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 390.531140] Lustre: 25115:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.534024] Lustre: 25115:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 390.536794] Lustre: 25115:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.539245] Lustre: 25115:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 390.542100] Lustre: 25115:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 393.594730] Lustre: 25086:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 393.597178] Lustre: 25086:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 393.599753] Lustre: 25086:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 393.602689] Lustre: 25086:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 393.605199] Lustre: 25086:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 393.607249] Lustre: 25086:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 393.609536] Lustre: 25086:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 393.613056] Lustre: 25086:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 393.616444] Lustre: 25086:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 393.619472] Lustre: 25086:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 393.622207] Lustre: 25086:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 393.624413] Lustre: 25086:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 400.113959] Lustre: 25087:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 400.116343] Lustre: 25087:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 400.118180] Lustre: 25087:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 400.120934] Lustre: 25087:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 400.122743] Lustre: 25087:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 400.124619] Lustre: 25087:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 400.126793] Lustre: 25087:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 400.129458] Lustre: 25087:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 400.132064] Lustre: 25087:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 400.134194] Lustre: 25087:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 400.136074] Lustre: 25087:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 400.138476] Lustre: 25087:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 409.714771] Lustre: 25086:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 409.718018] Lustre: 25086:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 409.720360] Lustre: 25086:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 409.723375] Lustre: 25086:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 409.725636] Lustre: 25086:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 409.727842] Lustre: 25086:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 409.729941] Lustre: 25086:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 409.732604] Lustre: 25086:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 409.734742] Lustre: 25086:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 409.737558] Lustre: 25086:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 409.740532] Lustre: 25086:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 409.743497] Lustre: 25086:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 451.040659] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 451.040795] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.048375] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 456.160802] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 461.281057] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 465.375115] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 465.425115] Lustre: server umount lustre-OST07c6 complete [ 468.423497] Lustre: server umount lustre-MDT0000 complete [ 473.935510] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 474.957566] Key type lgssc unregistered [ 475.076686] LNet: 28668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 475.078947] LNetError: 28668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 475.087279] LNet: Removed LNI 192.168.201.145@tcp [ 475.401671] Key type .llcrypt unregistered [ 475.403152] Key type ._llcrypt unregistered [ 482.839787] Key type ._llcrypt registered [ 482.840684] Key type .llcrypt registered [ 482.880821] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 483.207844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 483.240109] alg: No test for adler32 (adler32-zlib) [ 484.093989] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 484.189084] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 485.767201] Key type lgssc registered [ 486.104314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 488.584044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 488.593932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.731846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.073529] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 493.901574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.016835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 495.394827] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 497.925085] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 499.920342] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 502.115239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 502.234136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 504.281830] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 506.821441] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 514.535709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 524.769827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 524.771394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.774164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 524.778546] Lustre: Skipped 1 previous similar message [ 529.169991] Lustre: server umount lustre-OST0000 complete [ 535.008618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 535.008833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.009631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 535.009636] Lustre: Skipped 1 previous similar message [ 535.011408] LustreError: Skipped 1 previous similar message [ 535.016554] Lustre: Skipped 1 previous similar message [ 536.742794] Lustre: server umount lustre-MDT0000 complete [ 538.107318] LustreError: 29939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413417 with bad export cookie 3602155138123086292 [ 538.111573] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.112214] LustreError: 29939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 538.242209] Lustre: server umount lustre-MDT0001 complete [ 541.913368] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 542.929964] Key type lgssc unregistered [ 543.065628] LNet: 33667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.068718] LNetError: 33667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 543.076370] LNet: Removed LNI 192.168.201.145@tcp [ 543.386322] Key type .llcrypt unregistered [ 543.387361] Key type ._llcrypt unregistered [ 550.824016] Key type ._llcrypt registered [ 550.825474] Key type .llcrypt registered [ 550.865485] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 551.203371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 551.238821] alg: No test for adler32 (adler32-zlib) [ 552.096685] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 552.187400] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 553.767142] Key type lgssc registered [ 554.122630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 556.524778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 556.530325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.658713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.010018] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 561.800829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.925433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 563.415307] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 565.818767] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 567.687449] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 569.943704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 570.046559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.987118] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 574.628220] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 582.375590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 592.865447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 592.866442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.868757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 592.870876] Lustre: Skipped 1 previous similar message [ 595.792721] Lustre: server umount lustre-OST0000 complete [ 597.983983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 597.984650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.986289] LustreError: Skipped 1 previous similar message [ 597.990581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 597.992723] Lustre: Skipped 2 previous similar messages [ 603.104091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.105557] Lustre: Skipped 1 previous similar message [ 608.223921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 608.225502] Lustre: Skipped 1 previous similar message [ 611.295130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 611.350557] Lustre: server umount lustre-MDT0000 complete [ 612.641246] LustreError: 36754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413491 with bad export cookie 18219849538927497636 [ 612.644337] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.645944] LustreError: 36754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 612.756843] Lustre: server umount lustre-MDT0001 complete [ 616.190186] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 617.178453] Key type lgssc unregistered [ 617.291459] LNet: 38468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.294881] LNetError: 38468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 617.302247] LNet: Removed LNI 192.168.201.145@tcp [ 617.596388] Key type .llcrypt unregistered [ 617.597742] Key type ._llcrypt unregistered [ 625.911393] Key type ._llcrypt registered [ 625.912791] Key type .llcrypt registered [ 625.949836] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 626.249203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 626.290076] alg: No test for adler32 (adler32-zlib) [ 627.133351] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 627.219281] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 628.791143] Key type lgssc registered [ 629.132327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 631.822690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 631.828394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.000887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.374183] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 637.259636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.410215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 638.798351] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 641.389439] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 643.415866] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 645.816968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 645.960224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 647.977241] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 650.436987] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 658.277665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 663.010821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 663.013420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 663.017512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 663.521456] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 663.526906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 663.529478] Lustre: Skipped 1 previous similar message [ 665.875925] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 668.641331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 668.643283] Lustre: Skipped 2 previous similar messages [ 669.210583] Lustre: server umount lustre-MDT0000 complete [ 670.510446] LustreError: 40596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413549 with bad export cookie 13322436184522128512 [ 670.514077] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 670.516746] LustreError: 40596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 670.987326] LustreError: 39886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.993865] LustreError: 39886:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 673.759899] LustreError: 42348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 673.760137] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.769117] Lustre: Skipped 1 previous similar message [ 673.771355] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 673.773786] Lustre: Skipped 1 previous similar message [ 675.807936] LustreError: 39857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 676.719611] Lustre: server umount lustre-MDT0001 complete [ 694.539714] Lustre: server umount lustre-OST0000 complete [ 699.802152] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 700.801045] Key type lgssc unregistered [ 700.934361] LNet: 43739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.936365] LNetError: 43739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.948309] LNet: Removed LNI 192.168.201.145@tcp [ 701.242723] Key type .llcrypt unregistered [ 701.243846] Key type ._llcrypt unregistered [ 710.371226] Key type ._llcrypt registered [ 710.372364] Key type .llcrypt registered [ 710.410367] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 710.706630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 710.712742] alg: No test for adler32 (adler32-zlib) [ 711.557085] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 711.644632] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 713.223163] Key type lgssc registered [ 713.584301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 716.129465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 716.137177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 717.289792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 718.628848] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 721.386683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.496307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 722.736604] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 725.128680] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 726.979750] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 729.208764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 729.320535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 731.167499] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 733.553803] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 742.629755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 746.976654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 746.979317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 746.984616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 748.001916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 748.002050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 748.006583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.330452] Lustre: lustre-OST0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 749.333355] Lustre: Skipped 1 previous similar message [ 752.524903] Lustre: server umount lustre-OST0000 complete [ 765.282396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.288166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 765.290876] Lustre: Skipped 1 previous similar message [ 769.312142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 769.314055] Lustre: Skipped 1 previous similar message [ 771.480660] Lustre: server umount lustre-MDT0000 complete [ 772.710175] LustreError: 45114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413651 with bad export cookie 6249869874869422123 [ 772.713639] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 772.715197] LustreError: 45114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 778.860413] Lustre: server umount lustre-MDT0001 complete [ 782.409856] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 783.463623] Key type lgssc unregistered [ 783.604499] LNet: 48856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.607118] LNetError: 48856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.614444] LNet: Removed LNI 192.168.201.145@tcp [ 783.931084] Key type .llcrypt unregistered [ 783.932072] Key type ._llcrypt unregistered [ 792.337572] Key type ._llcrypt registered [ 792.338594] Key type .llcrypt registered [ 792.370883] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 792.685147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 792.712074] alg: No test for adler32 (adler32-zlib) [ 793.550158] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 793.633210] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 795.207137] Key type lgssc registered [ 795.566884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 797.953104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 797.958819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.094236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 800.348627] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 803.020227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.138486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 804.357589] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 806.603373] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 808.429319] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 810.701141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 810.808374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 812.827793] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 815.421576] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 824.101948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 828.141679] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 829.208564] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 839.891181] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-1818h28m38s [ 840.346650] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 841.488279] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 842.507092] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 844.255748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 844.257927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.261691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 847.187192] Lustre: server umount lustre-MDT0000 complete [ 848.345322] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 848.650841] LustreError: 50243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.657574] LustreError: 50243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 850.627966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 850.671668] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 850.770419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.798742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 851.817686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 852.184852] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 853.357390] Lustre: DEBUG MARKER: Wait for df (36564) ... [ 853.872931] Lustre: DEBUG MARKER: done [ 856.032156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 856.034466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.038222] Lustre: Skipped 2 previous similar messages [ 856.039969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 856.041775] Lustre: Skipped 5 previous similar messages [ 857.059076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 860.873342] Lustre: server umount lustre-OST0000 complete [ 862.176441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 862.176641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 862.182859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 862.184651] Lustre: Skipped 3 previous similar messages [ 868.315608] Lustre: server umount lustre-MDT0000 complete [ 869.591230] LustreError: 50230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413748 with bad export cookie 18078520678005026760 [ 869.593844] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 869.595041] LustreError: 50230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 869.708888] Lustre: server umount lustre-MDT0001 complete [ 873.304463] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 874.334943] Key type lgssc unregistered [ 874.455371] LNet: 55183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.457779] LNetError: 55183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.468294] LNet: Removed LNI 192.168.201.145@tcp [ 874.750075] Key type .llcrypt unregistered [ 874.751172] Key type ._llcrypt unregistered [ 888.634926] Key type ._llcrypt registered [ 888.635827] Key type .llcrypt registered [ 888.669250] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 889.066945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 889.073700] alg: No test for adler32 (adler32-zlib) [ 889.933453] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 890.015723] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 891.591154] Key type lgssc registered [ 891.931152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 894.362988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 894.368827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.448086] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 895.454473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 895.505495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.775915] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 899.399829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.429824] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 899.439994] Lustre: 57374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 899.525742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 900.909400] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 903.232637] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 904.967159] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 907.213425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 907.282775] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 907.344526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 909.265121] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 911.771958] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 911.890349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 915.211360] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-1818h29m54s [ 915.748736] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 918.336725] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 919.334373] Lustre: DEBUG MARKER: Creating a test file and stat it [ 919.379353] LustreError: 59896:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 939.796750] Lustre: lustre-MDT0000: Client fbcb5841-2887-4aa7-a4ef-69b4bfec1273 (at 192.168.201.45@tcp) reconnecting [ 959.423151] LustreError: 59896:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 960.183619] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 961.007311] Lustre: DEBUG MARKER: done [ 963.041980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.042709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 963.046728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 963.051403] Lustre: Skipped 1 previous similar message [ 968.161618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 968.163564] Lustre: Skipped 1 previous similar message [ 968.786422] Lustre: server umount lustre-OST0000 complete [ 971.231817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 971.234714] LustreError: Skipped 1 previous similar message [ 971.236564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.241240] Lustre: Skipped 1 previous similar message [ 971.243482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 976.212336] Lustre: server umount lustre-MDT0000 complete [ 977.431717] LustreError: 58114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413856 with bad export cookie 14738440387723746254 [ 977.434212] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 977.437419] LustreError: 58114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 977.549933] Lustre: server umount lustre-MDT0001 complete [ 981.101536] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 982.075690] Key type lgssc unregistered [ 982.205394] LNet: 61696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.208259] LNetError: 61696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.221332] LNet: Removed LNI 192.168.201.145@tcp [ 982.505052] Key type .llcrypt unregistered [ 982.506115] Key type ._llcrypt unregistered [ 997.999168] Key type ._llcrypt registered [ 998.000066] Key type .llcrypt registered [ 998.037343] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 998.462401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.472638] alg: No test for adler32 (adler32-zlib) [ 999.317913] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 999.396184] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1000.975144] Key type lgssc registered [ 1001.308195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1002.339269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1002.346502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.424720] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1003.432194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1003.486081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.803130] Lustre: Failing over lustre-MDT0000 [ 1003.892989] Lustre: server umount lustre-MDT0000 complete [ 1006.028302] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 18:31:25 (1784413885) [ 1008.154382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.277921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1009.546774] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1012.313171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.346275] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1012.357758] Lustre: 64470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1012.447470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1013.719036] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1016.166330] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1018.034278] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1020.260812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1020.335162] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1020.406621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.363767] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1023.433916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1024.878536] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1027.914219] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1033.183928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1033.185924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.189882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.697209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.698092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.700522] Lustre: Skipped 1 previous similar message [ 1035.988432] Lustre: server umount lustre-MDT0000 complete [ 1037.238970] LustreError: 64460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413916 with bad export cookie 9313360717323953290 [ 1037.242286] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1037.242775] LustreError: 64460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1037.332689] Lustre: server umount lustre-MDT0001 complete [ 1038.377868] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1039.884444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1042.906833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1046.943529] LustreError: 63392:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b7844568 x1871093566568448/t0(0) o250->MGC192.168.201.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1047.047496] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.089065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1047.120660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1048.503610] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1051.436286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.780356] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1054.628230] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1055.019301] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1055.647121] Lustre: 63393:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784413918/real 1784413918] req@000000007f87b02d x1871093566568064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784413934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1056.892389] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1059.551171] Lustre: 63394:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784413922/real 1784413922] req@000000003db79c3e x1871093566568192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784413938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1059.810755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.811793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1059.815512] Lustre: Skipped 2 previous similar messages [ 1059.817673] Lustre: Skipped 2 previous similar messages [ 1060.831200] Lustre: 63395:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784413923/real 1784413923] req@00000000669fa55b x1871093566568320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784413939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1063.647147] Lustre: 63396:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784413926/real 1784413926] req@000000003492219a x1871093566568576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784413942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1064.730590] Lustre: server umount lustre-MDT0000 complete [ 1064.928193] LustreError: 69719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.934044] LustreError: 69719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1066.015049] LustreError: 66321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413945 with bad export cookie 9313360717323955558 [ 1066.018196] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1066.019093] LustreError: 66321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1066.118599] Lustre: server umount lustre-MDT0001 complete [ 1068.218045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1071.581857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.191535] LustreError: 63392:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000444d0bdf x1871093566588032/t0(0) o250->MGC192.168.201.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1076.301086] LustreError: 68964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1076.325612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.327853] Lustre: Skipped 1 previous similar message [ 1076.350132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1077.669363] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1080.604130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.708607] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.160745] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1084.573780] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1085.855135] Lustre: 63395:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784413949/real 1784413949] req@0000000030f4db3c x1871093566587648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784413965 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1085.924181] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1085.927065] Lustre: Skipped 1 previous similar message [ 1086.449905] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1090.275511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.276617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1090.280565] Lustre: Skipped 1 previous similar message [ 1090.283409] Lustre: Skipped 2 previous similar messages [ 1094.352786] Lustre: server umount lustre-MDT0000 complete [ 1095.648932] LustreError: 70911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.654235] LustreError: 70911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1095.683211] LustreError: 66321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784413974 with bad export cookie 9313360717323956937 [ 1095.686677] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1095.688278] LustreError: 66321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1095.785658] Lustre: server umount lustre-MDT0001 complete [ 1097.900217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1098.980701] Lustre: DEBUG MARKER: files compared the same [ 1108.887724] Lustre: server umount lustre-OST0000 complete [ 1114.620523] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1115.708329] Key type lgssc unregistered [ 1115.824353] LNet: 74181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.826308] LNetError: 74181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.836276] LNet: Removed LNI 192.168.201.145@tcp [ 1116.132749] Key type .llcrypt unregistered [ 1116.133716] Key type ._llcrypt unregistered [ 1125.234395] Key type ._llcrypt registered [ 1125.236010] Key type .llcrypt registered [ 1125.271500] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1125.606455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1125.664366] alg: No test for adler32 (adler32-zlib) [ 1126.521813] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1126.618355] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1128.199140] Key type lgssc registered [ 1128.568566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1131.227129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1131.237027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.375255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1133.701307] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug +malloc all [ 1136.705814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1136.806667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1138.058887] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug +malloc all [ 1140.268100] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1142.116823] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1144.309779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1144.413526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.379951] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug +malloc all [ 1148.862643] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1157.730987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1163.232054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1163.232153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.238234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1163.239737] Lustre: Skipped 1 previous similar message [ 1163.975743] Lustre: server umount lustre-OST0000 complete [ 1167.247429] Lustre: server umount lustre-MDT0000 complete [ 1167.839399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1167.841865] LustreError: Skipped 1 previous similar message [ 1167.842928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.846175] Lustre: Skipped 1 previous similar message [ 1167.847392] LustreError: 75574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.852164] LustreError: 75574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1168.351524] LustreError: 78058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1168.360053] LustreError: 78058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1168.426978] LustreError: 75555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414047 with bad export cookie 885664507332989700 [ 1168.429097] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1168.430909] LustreError: 75555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1168.536210] Lustre: server umount lustre-MDT0001 complete [ 1171.897032] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1172.907788] Key type lgssc unregistered [ 1173.030306] LNet: 79123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.033076] LNetError: 79123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1173.044324] LNet: Removed LNI 192.168.201.145@tcp [ 1173.355792] Key type .llcrypt unregistered [ 1173.357063] Key type ._llcrypt unregistered [ 1181.699399] Key type ._llcrypt registered [ 1181.700250] Key type .llcrypt registered [ 1181.731362] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1182.049240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1182.074874] alg: No test for adler32 (adler32-zlib) [ 1182.916558] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1182.991960] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1184.559116] Key type lgssc registered [ 1184.925893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1187.538963] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1187.545948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1202.975427] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1208.095289] LustreError: 80489:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.145@tcp: failed processing log, type 4: rc = -110 [ 1234.719152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.703848] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1238.897425] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1241.238456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.276866] Lustre: *** cfs_fail_loc=706, val=0*** [ 1241.387949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.409501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1242.655061] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1245.283687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.398875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1246.647319] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1248.907055] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1250.664278] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1251.807883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1251.810291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.814195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1254.882158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1254.882911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.419467] Lustre: server umount lustre-OST0000 complete [ 1260.000789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.001757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1260.004725] Lustre: Skipped 1 previous similar message [ 1260.009978] Lustre: Skipped 2 previous similar messages [ 1264.856205] Lustre: server umount lustre-MDT0000 complete [ 1265.120947] LustreError: 81577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1265.126967] LustreError: 81577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1266.088258] LustreError: 80495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414145 with bad export cookie 17699397890311984880 [ 1266.090646] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1266.092311] LustreError: 80495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1266.210379] Lustre: server umount lustre-MDT0001 complete [ 1269.661358] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1270.697227] Key type lgssc unregistered [ 1270.823405] LNet: 84098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.827204] LNetError: 84098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.836297] LNet: Removed LNI 192.168.201.145@tcp [ 1271.139181] Key type .llcrypt unregistered [ 1271.140259] Key type ._llcrypt unregistered [ 1280.607495] Key type ._llcrypt registered [ 1280.609037] Key type .llcrypt registered [ 1280.648185] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1280.952827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1280.961264] alg: No test for adler32 (adler32-zlib) [ 1281.826578] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1281.926662] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1283.511151] Key type lgssc registered [ 1283.897579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1286.450458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1286.455785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1288.678976] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1290.978059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.138658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1292.335724] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1294.981148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1295.080706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1296.896574] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1299.604244] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1299.607729] Lustre: MGS: Client 96ef99d7-2ea8-4b6d-a7f0-d8a29df461ff (at 0@lo) reconnecting [ 1299.690303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1299.715886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1300.879341] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1310.176173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1310.178651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.182947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1312.224454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1312.224918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1312.226495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1314.635475] Lustre: server umount lustre-OST0000 complete [ 1317.344676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.345022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1317.345486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1317.345491] Lustre: Skipped 1 previous similar message [ 1317.349266] Lustre: Skipped 1 previous similar message [ 1322.463951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.465853] Lustre: Skipped 2 previous similar messages [ 1327.584747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1327.587625] Lustre: Skipped 1 previous similar message [ 1330.143111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1330.187772] Lustre: server umount lustre-MDT0000 complete [ 1331.537588] Lustre: server umount lustre-MDT0001 complete [ 1332.818606] Lustre: server umount MGS complete [ 1336.516043] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:36:55 (1784414215) [ 1345.661034] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 1348.376973] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1358.877415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1360.122063] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1363.980288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.071363] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1364.089495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1364.131391] Lustre: lustre-MDT0001: new disk, initializing [ 1364.156586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1364.167212] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1364.169392] Lustre: 93349:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1364.171849] LustreError: 93349:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1365.586347] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1369.769561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1369.856300] Lustre: lustre-OST0000: new disk, initializing [ 1369.857635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1369.859651] Lustre: 94115:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1369.877706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.732178] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1372.767980] Lustre: 94471:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000037a40a43 x1871093863318528/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784414268 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1375.329017] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1377.485480] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1377.489768] Lustre: MGS: Client 64487c85-5d99-42ee-a7b6-b01e31bb5a34 (at 0@lo) reconnecting [ 1377.503954] Lustre: 95161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1377.510026] LustreError: 93334:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1377.514224] LustreError: 93334:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1377.519752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1377.555365] Lustre: lustre-MDT0000: new disk, initializing [ 1377.600080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1378.655812] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1378.656047] Lustre: 95177:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d5e7dd47 x1871093863324288/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784414273 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1378.663565] Lustre: 95177:0:(client.c:1639:after_reply()) Skipped 1 previous similar message [ 1378.879897] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1381.857464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1381.859896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1383.713851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1383.716783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1383.717192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1404.383957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1404.386113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1404.389886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1417.183143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1417.247403] Lustre: server umount lustre-OST0000 complete [ 1422.305408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1422.305910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.308619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1422.311909] Lustre: Skipped 2 previous similar messages [ 1422.313776] Lustre: Skipped 6 previous similar messages [ 1424.711816] Lustre: server umount lustre-MDT0000 complete [ 1440.223117] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1440.307978] Lustre: server umount lustre-MDT0001 complete [ 1441.625506] Lustre: server umount MGS complete [ 1445.466598] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:38:44 (1784414324) [ 1450.995498] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1451.963595] Key type lgssc unregistered [ 1452.076398] LNet: 97681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.078531] LNetError: 97681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.087239] LNet: Removed LNI 192.168.201.145@tcp [ 1452.380815] Key type .llcrypt unregistered [ 1452.382122] Key type ._llcrypt unregistered [ 1459.047390] Key type ._llcrypt registered [ 1459.048314] Key type .llcrypt registered [ 1459.088727] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1459.548633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1459.580286] alg: No test for adler32 (adler32-zlib) [ 1460.439893] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1460.532819] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1462.111180] Key type lgssc registered [ 1462.479404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1464.853938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1464.860386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1466.011196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1467.317373] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1469.899552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1469.999090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1471.147801] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1473.387242] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1475.082909] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1477.194706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1477.296765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.072974] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1481.418333] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1491.622486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1496.031863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1496.034696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.039114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1497.057885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1497.058856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.059345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1500.307790] Lustre: server umount lustre-OST0000 complete [ 1502.177151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1502.177905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1502.180938] Lustre: Skipped 1 previous similar message [ 1502.182975] Lustre: Skipped 1 previous similar message [ 1507.296150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.298503] Lustre: Skipped 2 previous similar messages [ 1507.732654] Lustre: server umount lustre-MDT0000 complete [ 1508.933123] LustreError: 98859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414388 with bad export cookie 15127470300354349815 [ 1508.936451] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1508.938800] LustreError: 98859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1509.057415] Lustre: server umount lustre-MDT0001 complete [ 1512.884542] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1513.935313] Key type lgssc unregistered [ 1514.060487] LNet: 102429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1514.063820] LNetError: 102429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1514.073290] LNet: Removed LNI 192.168.201.145@tcp [ 1514.402224] Key type .llcrypt unregistered [ 1514.403253] Key type ._llcrypt unregistered [ 1521.216766] Key type ._llcrypt registered [ 1521.217754] Key type .llcrypt registered [ 1521.253538] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1521.637175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1521.672821] alg: No test for adler32 (adler32-zlib) [ 1522.526314] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1522.618797] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1524.207123] Key type lgssc registered [ 1524.612500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1526.883470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1526.885395] LustreError: 103487:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1526.893542] LustreError: 103490:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1526.897654] LustreError: 103487:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 1526.899885] LustreError: 103487:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1526.903935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1526.904371] LustreError: 103487:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1528.069782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1530.797382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1530.894979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1532.373279] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1534.118904] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing lsmod [ 1535.166818] LustreError: 104693:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1535.166841] LustreError: 104694:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1535.173983] LustreError: 104693:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1535.176350] LustreError: 104693:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1535.180071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1535.181527] LustreError: 104693:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1535.318310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.263776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1536.265277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1536.267551] Lustre: Skipped 1 previous similar message [ 1541.408909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1541.411170] Lustre: Skipped 4 previous similar messages [ 1542.423524] Lustre: server umount lustre-MDT0000 complete [ 1543.697229] LustreError: 103504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414422 with bad export cookie 3463410136422032074 [ 1543.698356] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1543.701080] LustreError: 103504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1543.707276] Lustre: 103519:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 1545.503291] LustreError: 103959:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002e683880 x1871094114958720/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1545.553505] Lustre: server umount lustre-MDT0001 complete [ 1557.075914] Lustre: server umount lustre-OST0000 complete [ 1559.788561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1559.928375] LustreError: 105757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1559.935282] LustreError: 105757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1559.952325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.286592] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1564.090842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.209138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1565.523011] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1567.864625] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1569.635868] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1571.857987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1573.828380] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1576.155263] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1585.253689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1589.728609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1589.731585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1589.737451] Lustre: Skipped 1 previous similar message [ 1589.740093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1590.752525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1590.754580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1593.878275] Lustre: server umount lustre-OST0000 complete [ 1595.872494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.873407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1595.878472] Lustre: Skipped 1 previous similar message [ 1595.882262] Lustre: Skipped 3 previous similar messages [ 1600.993410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1600.996125] Lustre: Skipped 1 previous similar message [ 1601.365931] Lustre: server umount lustre-MDT0000 complete [ 1602.612464] LustreError: 105738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414481 with bad export cookie 3463410136422032753 [ 1602.615464] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1602.617965] LustreError: 105738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1602.736440] Lustre: server umount lustre-MDT0001 complete [ 1606.561759] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1607.632883] Key type lgssc unregistered [ 1607.761397] LNet: 109309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1607.763590] LNetError: 109309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.771272] LNet: Removed LNI 192.168.201.145@tcp [ 1608.097368] Key type .llcrypt unregistered [ 1608.098686] Key type ._llcrypt unregistered [ 1616.826981] Key type ._llcrypt registered [ 1616.827868] Key type .llcrypt registered [ 1616.861452] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1617.163126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1617.203862] alg: No test for adler32 (adler32-zlib) [ 1618.057626] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1618.136480] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1619.711117] Key type lgssc registered [ 1620.036636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1622.473307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1622.477817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1623.602937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.887137] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1627.502236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1627.611066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1628.931087] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1631.247747] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1632.982538] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1635.109597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1635.217065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.062683] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1639.263323] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1649.574427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1651.870891] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1652.588933] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1654.756698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1654.758161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1654.760154] Lustre: Skipped 1 previous similar message [ 1654.765169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1655.565226] Lustre: server umount lustre-OST0000 complete [ 1658.335940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1658.340055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1658.345386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1658.347248] Lustre: Skipped 1 previous similar message [ 1658.973283] Lustre: server umount lustre-MDT0000 complete [ 1659.872504] LustreError: 110725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1659.872722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.879348] LustreError: 110725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1660.204157] LustreError: 110732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414539 with bad export cookie 10451066131907066308 [ 1660.204736] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1660.210502] LustreError: 110732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1660.323958] Lustre: server umount lustre-MDT0001 complete [ 1664.059740] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1665.106556] Key type lgssc unregistered [ 1665.233543] LNet: 114355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.236560] LNetError: 114355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.247451] LNet: Removed LNI 192.168.201.145@tcp [ 1665.599182] Key type .llcrypt unregistered [ 1665.600163] Key type ._llcrypt unregistered [ 1672.407441] Key type ._llcrypt registered [ 1672.408364] Key type .llcrypt registered [ 1672.442802] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1672.771703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1672.832473] alg: No test for adler32 (adler32-zlib) [ 1673.671159] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1673.747632] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1675.319136] Key type lgssc registered [ 1675.716412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1678.316140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1678.321892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.465516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1680.823702] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1683.663618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.788526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1685.223172] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1687.589460] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1689.315931] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1691.549980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1691.672808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1693.566819] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1695.932882] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1703.975188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1709.025373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1709.026150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.027252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1709.034188] Lustre: Skipped 1 previous similar message [ 1710.861419] Lustre: server umount lustre-OST0000 complete [ 1714.144733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.146350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1714.148564] Lustre: Skipped 1 previous similar message [ 1714.150413] Lustre: Skipped 1 previous similar message [ 1718.226187] Lustre: server umount lustre-MDT0000 complete [ 1719.264058] LustreError: 116285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1719.268603] LustreError: 116285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1719.478564] LustreError: 118054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414598 with bad export cookie 1900838117135813997 [ 1719.482861] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1719.598358] Lustre: server umount lustre-MDT0001 complete [ 1723.093664] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1724.071045] Key type lgssc unregistered [ 1724.195417] LNet: 119105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.197905] LNetError: 119105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.206293] LNet: Removed LNI 192.168.201.145@tcp [ 1724.523033] Key type .llcrypt unregistered [ 1724.524730] Key type ._llcrypt unregistered [ 1733.021588] Key type ._llcrypt registered [ 1733.022590] Key type .llcrypt registered [ 1733.062667] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1733.634361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1733.647902] alg: No test for adler32 (adler32-zlib) [ 1734.497683] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1734.587659] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1736.167115] Key type lgssc registered [ 1736.543585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1739.116930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1739.125293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1740.275870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1741.664642] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1744.582287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.702686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1746.114565] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1748.440595] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1750.279190] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1752.459559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1752.608916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.489595] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1756.879296] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1764.899253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1770.149888] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1771.413758] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1771.416074] Lustre: Skipped 2 previous similar messages [ 1772.827065] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1772.830496] Lustre: Skipped 2 previous similar messages [ 1778.688914] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1778.702900] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1781.666557] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.45@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1781.670896] Lustre: Skipped 2 previous similar messages [ 1788.917094] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.45@tcp 192.168.0.[2,10]@tcp [ 1790.819548] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.45/24@tcp in log lustre-MDT0000 [ 1790.822672] Lustre: Skipped 2 previous similar messages [ 1797.631590] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.45/24@tcp [ 1797.634854] Lustre: Skipped 1 previous similar message [ 1800.676161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1800.676460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.682713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1800.685391] Lustre: Skipped 1 previous similar message [ 1805.007447] Lustre: server umount lustre-OST0000 complete [ 1806.303820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1806.305864] LustreError: Skipped 1 previous similar message [ 1806.306988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1806.310085] Lustre: Skipped 1 previous similar message [ 1806.311534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1810.912461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1810.913586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1812.313824] Lustre: server umount lustre-MDT0000 complete [ 1813.566317] LustreError: 120480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414692 with bad export cookie 3994285081661624796 [ 1813.569255] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1813.572135] LustreError: 120480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1813.686587] Lustre: server umount lustre-MDT0001 complete [ 1817.336032] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1818.362582] Key type lgssc unregistered [ 1818.484403] LNet: 125835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.487554] LNetError: 125835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.494289] LNet: Removed LNI 192.168.201.145@tcp [ 1818.808778] Key type .llcrypt unregistered [ 1818.810159] Key type ._llcrypt unregistered [ 1829.050127] Key type ._llcrypt registered [ 1829.051664] Key type .llcrypt registered [ 1829.091158] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1829.471954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1829.508966] alg: No test for adler32 (adler32-zlib) [ 1830.359267] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1830.450436] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1832.031138] Key type lgssc registered [ 1832.400277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1834.839074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1834.844044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.958978] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1835.960206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.965582] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.45/24@tcp [ 1837.188789] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1839.773058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1839.882405] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1839.891358] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.45/24@tcp [ 1839.906893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1841.271023] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1843.655783] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1845.439415] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1847.525662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1847.632391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.405322] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1851.639594] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1860.967592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 1866.208508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1866.212128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.213669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.217833] Lustre: Skipped 1 previous similar message [ 1867.541636] Lustre: server umount lustre-OST0000 complete [ 1870.815868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1870.818393] LustreError: Skipped 1 previous similar message [ 1870.820320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.824864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1870.826699] Lustre: Skipped 1 previous similar message [ 1870.935966] Lustre: server umount lustre-MDT0000 complete [ 1871.328561] LustreError: 129688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.333123] LustreError: 129688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1872.206250] LustreError: 127352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414751 with bad export cookie 1680105679566283715 [ 1872.208865] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1872.209970] LustreError: 127352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1872.325653] Lustre: server umount lustre-MDT0001 complete [ 1876.013579] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1877.150727] Key type lgssc unregistered [ 1877.281403] LNet: 130975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.284902] LNetError: 130975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.292286] LNet: Removed LNI 192.168.201.145@tcp [ 1877.593631] Key type .llcrypt unregistered [ 1877.594706] Key type ._llcrypt unregistered [ 1886.297686] Key type ._llcrypt registered [ 1886.298689] Key type .llcrypt registered [ 1886.337395] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 1892.671270] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1893.061614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1893.092150] alg: No test for adler32 (adler32-zlib) [ 1893.945259] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 1894.033545] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1895.615153] Key type lgssc registered [ 1896.055334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1907.406918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1907.412132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.494228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1908.507222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1908.541556] Lustre: lustre-MDT0000: new disk, initializing [ 1908.561832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.567657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1909.897524] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1914.115281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.163170] Lustre: 134347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1914.191331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1914.193041] Lustre: Skipped 1 previous similar message [ 1914.228463] Lustre: lustre-MDT0001: new disk, initializing [ 1914.268984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1914.282793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1914.286404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1915.777856] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1919.466524] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1921.145655] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1923.259768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1923.355987] Lustre: lustre-OST0000: new disk, initializing [ 1923.358222] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1923.362579] Lustre: 136316:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1923.368544] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1923.399082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.083312] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1928.650467] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1928.682187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1928.686081] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1928.699444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1930.106103] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1930.171125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1931.546505] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1931.618346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1933.793677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1933.794837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1933.795540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1933.796570] LustreError: Skipped 1 previous similar message [ 1933.800441] Lustre: Skipped 1 previous similar message [ 1934.290151] Lustre: server umount lustre-OST0000 complete [ 1938.913742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1938.914663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1938.917340] Lustre: Skipped 1 previous similar message [ 1938.920842] Lustre: Skipped 2 previous similar messages [ 1941.657397] Lustre: server umount lustre-MDT0000 complete [ 1942.867192] LustreError: 134340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414822 with bad export cookie 15967648140925694565 [ 1942.871033] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1942.873471] LustreError: 134340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1942.984123] Lustre: server umount lustre-MDT0001 complete [ 1945.850398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.029612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.351412] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1950.104447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.222411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1951.580461] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1954.038881] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1955.830654] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1957.990879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1959.966531] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1962.260312] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1963.599827] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1971.826435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 1973.207672] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1973.287654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1976.738537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1976.821859] Lustre: lustre-OST0001: new disk, initializing [ 1976.823476] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1976.825642] Lustre: 141953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1976.855222] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1976.857552] Lustre: Skipped 1 previous similar message [ 1978.965753] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1982.374880] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1983.489791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1983.494645] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1983.521488] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1984.500158] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1985.915791] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1985.995167] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1987.912477] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9598468c1000.ost_server_uuid 50 [ 1988.435582] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9598468c1000.ost_server_uuid in FULL state after 0 sec [ 1993.705126] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1993.705644] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.714227] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1993.716099] Lustre: Skipped 1 previous similar message [ 1996.479173] Lustre: server umount lustre-OST0001 complete [ 1998.818693] LustreError: 140348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.819219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.820272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1998.825979] LustreError: 140348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1998.829541] Lustre: Skipped 2 previous similar messages [ 2003.939195] LustreError: 140348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.943665] LustreError: 140348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2009.057870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2009.058695] LustreError: 140348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.059750] Lustre: Skipped 4 previous similar messages [ 2009.064805] LustreError: 140348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2012.127174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2012.184874] Lustre: server umount lustre-OST0000 complete [ 2014.178170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.680299] Lustre: server umount lustre-MDT0000 complete [ 2020.989825] LustreError: 141950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784414900 with bad export cookie 15967648140925695433 [ 2020.995087] LustreError: 141950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2020.995719] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.123381] Lustre: server umount lustre-MDT0001 complete [ 2027.597579] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2028.665232] Key type lgssc unregistered [ 2028.786331] LNet: 144971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.789768] LNetError: 144971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.796314] LNet: Removed LNI 192.168.201.145@tcp [ 2029.109636] Key type .llcrypt unregistered [ 2029.110625] Key type ._llcrypt unregistered [ 2042.314657] Key type ._llcrypt registered [ 2042.315600] Key type .llcrypt registered [ 2042.355690] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 2048.706791] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2049.081546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2049.119265] alg: No test for adler32 (adler32-zlib) [ 2049.971544] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2050.065287] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2051.655156] Key type lgssc registered [ 2052.078449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2063.851239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2063.860237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.938702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2064.948319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2064.985804] Lustre: lustre-MDT0000: new disk, initializing [ 2065.006319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2065.012777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2066.330600] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2070.612952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.657354] Lustre: 149094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2070.684307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2070.686972] Lustre: Skipped 1 previous similar message [ 2070.721250] Lustre: lustre-MDT0001: new disk, initializing [ 2070.753243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2070.764013] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2070.766842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2072.075441] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2075.794413] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2077.629343] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2079.863353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2079.962845] Lustre: lustre-OST0000: new disk, initializing [ 2079.964713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2079.967496] Lustre: 151064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2079.997428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.911446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2081.914756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2081.951744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2081.961222] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2085.783200] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2087.828872] Lustre: Failing over lustre-OST0000 [ 2087.900515] Lustre: server umount lustre-OST0000 complete [ 2100.785734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2100.914156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.919470] Lustre: Skipped 1 previous similar message [ 2100.923595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.933883] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2101.718929] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2103.019127] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2104.325608] Lustre: Failing over lustre-MDT0000 [ 2104.435447] Lustre: server umount lustre-MDT0000 complete [ 2106.336962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.342304] Lustre: Skipped 2 previous similar messages [ 2108.383183] Lustre: 147325:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784414971/real 1784414971] req@00000000e643e56c x1871094668617856/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784414987 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2111.456061] LustreError: 149907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2111.462104] LustreError: 149907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2113.503143] Lustre: 147324:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784414976/real 1784414976] req@0000000009b8cb6a x1871094668619648/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784414992 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2116.577118] LustreError: 149106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2116.583760] LustreError: 149106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2117.524512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.568563] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2117.573084] Lustre: 149099:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2117.659200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2117.682623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.716497] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2119.076380] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2121.995597] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2122.730736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2122.735637] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2122.755095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2127.845753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.847036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2127.847688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2127.849215] Lustre: Skipped 1 previous similar message [ 2127.853081] LustreError: Skipped 1 previous similar message [ 2129.869164] Lustre: server umount lustre-OST0000 complete [ 2132.961649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2132.962267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.963533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2132.963539] Lustre: Skipped 1 previous similar message [ 2132.970919] Lustre: Skipped 1 previous similar message [ 2137.306717] Lustre: server umount lustre-MDT0000 complete [ 2138.079947] LustreError: 149107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.083958] LustreError: 149107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2138.548116] LustreError: 149890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415017 with bad export cookie 3635497224597743632 [ 2138.552236] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2138.552925] LustreError: 149890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2138.661964] Lustre: server umount lustre-MDT0001 complete [ 2142.388962] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2143.492973] Key type lgssc unregistered [ 2143.615350] LNet: 154706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.617741] LNetError: 154706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2143.626249] LNet: Removed LNI 192.168.201.145@tcp [ 2143.911824] Key type .llcrypt unregistered [ 2143.912723] Key type ._llcrypt unregistered [ 2163.112905] Key type ._llcrypt registered [ 2163.114556] Key type .llcrypt registered [ 2163.155940] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2163.606356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2163.614866] alg: No test for adler32 (adler32-zlib) [ 2164.461319] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2164.551118] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2166.135123] Key type lgssc registered [ 2166.525332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2169.223649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2169.229286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.378064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.706282] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2175.516713] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2177.536160] LustreError: 156252:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2177.541173] LustreError: 156252:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2177.544269] LustreError: 156252:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2177.679576] Lustre: server umount lustre-MDT0000 complete [ 2182.755658] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 2186.086569] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2198.000629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.107470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2198.120506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2198.161885] Lustre: lustre-MDT0000: new disk, initializing [ 2198.185615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2198.192088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2199.719485] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2203.682979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.720729] Lustre: 159548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2203.743638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2203.745457] Lustre: Skipped 1 previous similar message [ 2203.778400] Lustre: lustre-MDT0001: new disk, initializing [ 2203.810227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2203.821149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2203.824089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2205.081561] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2208.556056] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2210.266629] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2212.350823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2212.448117] Lustre: lustre-OST0000: new disk, initializing [ 2212.449826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2212.452672] Lustre: 161515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2212.479908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.197813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2214.200184] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2214.230115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2214.328338] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2217.878513] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2219.640268] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2220.147069] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:51:39 (1784415099) [ 2286.047753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.048031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2286.048149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2286.052114] Lustre: Skipped 2 previous similar messages [ 2291.167448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2291.169368] Lustre: Skipped 4 previous similar messages [ 2296.287117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2296.287718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2296.291797] Lustre: Skipped 2 previous similar messages [ 2296.353415] Lustre: server umount lustre-MDT0000 complete [ 2297.638178] LustreError: 159541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415176 with bad export cookie 7166136571003976033 [ 2297.639599] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2297.641650] LustreError: 159541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2301.407586] LustreError: 159554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.407614] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.411382] LustreError: 159554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2301.419643] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2302.431478] LustreError: 162235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.779677] Lustre: server umount lustre-MDT0001 complete [ 2319.327185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2319.384949] Lustre: server umount lustre-OST0000 complete [ 2322.509382] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 2325.579477] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2337.405020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.498687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2337.509656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2337.551145] Lustre: lustre-MDT0000: new disk, initializing [ 2337.572435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.578937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2338.910437] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2343.018043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2343.062063] Lustre: 166143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2343.089181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2343.091091] Lustre: Skipped 1 previous similar message [ 2343.130111] Lustre: lustre-MDT0001: new disk, initializing [ 2343.177449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2343.181872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2344.577747] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2348.272676] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2350.133841] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2352.439266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2352.555952] Lustre: lustre-OST0000: new disk, initializing [ 2352.557593] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2352.560804] Lustre: 168112:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2352.589896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2352.591751] Lustre: Skipped 1 previous similar message [ 2354.498497] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2358.182557] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2358.775464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2358.778454] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2358.814628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2359.679707] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2359.753273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2361.243327] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2361.324139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2363.875440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2363.875599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.877711] LustreError: Skipped 1 previous similar message [ 2363.878242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2363.882131] Lustre: Skipped 1 previous similar message [ 2363.886733] Lustre: Skipped 2 previous similar messages [ 2368.084635] Lustre: server umount lustre-OST0000 complete [ 2374.113392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2374.113466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.114638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2374.121518] Lustre: Skipped 1 previous similar message [ 2375.459512] Lustre: server umount lustre-MDT0000 complete [ 2376.691446] LustreError: 166136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415255 with bad export cookie 7166136571004134849 [ 2376.694340] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2376.694754] LustreError: 166136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2376.817613] Lustre: server umount lustre-MDT0001 complete [ 2379.525545] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:54:18 (1784415258) [ 2384.659614] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 2387.609379] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2399.132770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2399.230177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2399.240659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2399.285662] Lustre: lustre-MDT0000: new disk, initializing [ 2399.306611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2399.312679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2400.616454] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2404.616532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2404.665219] Lustre: 172861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2404.777661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2406.004205] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2409.555671] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2411.260333] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2413.562410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2413.699942] Lustre: 174836:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2415.774413] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2419.392224] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2420.737164] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2420.788960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2430.944973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.946758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.947994] LustreError: Skipped 1 previous similar message [ 2430.952714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.955819] Lustre: Skipped 2 previous similar messages [ 2432.400963] Lustre: server umount lustre-OST0000 complete [ 2435.551939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2439.777491] Lustre: server umount lustre-MDT0000 complete [ 2441.013265] LustreError: 172852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415320 with bad export cookie 7166136571004135703 [ 2441.016874] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2441.018342] LustreError: 172852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2441.126970] Lustre: server umount lustre-MDT0001 complete [ 2443.952784] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:55:23 (1784415323) [ 2449.135310] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 2452.145446] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2463.575328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.659650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2463.670492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2463.672890] Lustre: Skipped 3 previous similar messages [ 2463.708522] Lustre: lustre-MDT0000: new disk, initializing [ 2463.710324] Lustre: Skipped 2 previous similar messages [ 2463.731783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.733592] Lustre: Skipped 2 previous similar messages [ 2463.738738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2463.742148] Lustre: Skipped 2 previous similar messages [ 2465.135557] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2469.333768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.380537] Lustre: 179149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2469.481589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2470.879885] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2474.665135] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2476.617190] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2478.908026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2479.022194] Lustre: 181125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2481.005353] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2483.755597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2484.783829] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2488.802264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2488.804328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.805425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2488.807184] Lustre: Skipped 4 previous similar messages [ 2488.808751] Lustre: Skipped 3 previous similar messages [ 2492.750062] Lustre: server umount lustre-OST0000 complete [ 2508.255093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2508.322287] Lustre: server umount lustre-MDT0000 complete [ 2509.281135] LustreError: 179161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.290501] LustreError: 179161:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2509.602518] LustreError: 179144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415388 with bad export cookie 7166136571004136774 [ 2509.607089] LustreError: 179144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2509.607712] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2509.747536] Lustre: server umount lustre-MDT0001 complete [ 2513.646186] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2514.641302] Key type lgssc unregistered [ 2514.775349] LNet: 182981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2514.777398] LNetError: 182981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2514.788328] LNet: Removed LNI 192.168.201.145@tcp [ 2515.090234] Key type .llcrypt unregistered [ 2515.091352] Key type ._llcrypt unregistered [ 2523.890993] Key type ._llcrypt registered [ 2523.891899] Key type .llcrypt registered [ 2523.924943] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2524.304972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2524.342752] alg: No test for adler32 (adler32-zlib) [ 2525.211548] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2525.307130] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2526.895231] Key type lgssc registered [ 2527.371386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2529.946966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2529.952490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.094817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2532.440676] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2535.207682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2535.333499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2536.641522] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2538.975389] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2540.767984] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2542.928517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2543.058151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.960711] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2547.408618] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2556.395520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2570.719925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2570.722044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.725689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2571.746367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2571.746443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2571.747468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.749131] Lustre: Skipped 1 previous similar message [ 2575.507330] Lustre: server umount lustre-OST0000 complete [ 2581.472287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2581.475241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.479541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.104340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.106390] Lustre: Skipped 2 previous similar messages [ 2591.199152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2591.270669] Lustre: server umount lustre-MDT0000 complete [ 2592.223957] LustreError: 184374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.229292] LustreError: 184374:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2592.477137] LustreError: 184356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415471 with bad export cookie 1958921743559129804 [ 2592.480253] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2592.482548] LustreError: 184356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2592.600058] Lustre: server umount lustre-MDT0001 complete [ 2596.131122] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 2597.172923] Key type lgssc unregistered [ 2597.304466] LNet: 187924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2597.308109] LNetError: 187924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2597.315278] LNet: Removed LNI 192.168.201.145@tcp [ 2597.621653] Key type .llcrypt unregistered [ 2597.623113] Key type ._llcrypt unregistered [ 2606.841399] Key type ._llcrypt registered [ 2606.842888] Key type .llcrypt registered [ 2606.886102] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2607.302552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2607.345322] alg: No test for adler32 (adler32-zlib) [ 2608.213312] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 2608.306955] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 2609.887107] Key type lgssc registered [ 2610.257701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2612.997308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2613.003198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2614.162071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2615.510862] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2618.269632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2618.399486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2619.782387] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2622.150681] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2623.965531] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2626.239227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2626.365616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2628.360093] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2630.766700] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2639.722129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2643.936592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2643.939160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2643.944454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2644.962867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2644.963567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2644.964462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2646.293189] Lustre: lustre-OST0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 2646.295521] Lustre: Skipped 1 previous similar message [ 2649.428130] Lustre: server umount lustre-OST0000 complete [ 2651.824358] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9598476f1800.ost_server_uuid 50 [ 2652.373190] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2662.339859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.340926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2662.343130] Lustre: Skipped 1 previous similar message [ 2666.400228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2666.402110] Lustre: Skipped 2 previous similar messages [ 2668.504707] Lustre: server umount lustre-MDT0000 complete [ 2669.726345] LustreError: 189299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415548 with bad export cookie 11494688748912485555 [ 2669.729848] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2669.730576] LustreError: 189299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2669.858677] Lustre: server umount lustre-MDT0001 complete [ 2672.550119] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:59:11 (1784415551) [ 2674.782807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.949905] LustreError: 193227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.954905] LustreError: 193227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2674.977464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2676.281936] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2679.112270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.243947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2680.635139] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2683.170951] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2684.985710] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2687.674516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2687.829551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.695515] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2692.207325] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2694.545434] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2694.619303] Lustre: lustre-OST0001: new disk, initializing [ 2694.620929] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2694.623422] Lustre: 196064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2696.538268] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2697.540551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2697.546045] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2697.574080] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2700.239925] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2700.648467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2704.351905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2704.354323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.358532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2707.198900] Lustre: server umount lustre-OST0000 complete [ 2709.248351] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2709.329846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2710.667457] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2710.733697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2710.796484] LustreError: 195045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.917927] LustreError: 195047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.927340] LustreError: 195047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2721.250960] LustreError: 195047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.253261] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.253745] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2721.253749] Lustre: Skipped 3 previous similar messages [ 2721.254759] LustreError: 195047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2721.258637] Lustre: Skipped 2 previous similar messages [ 2726.370750] LustreError: 195046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.376047] LustreError: 195046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2726.866202] Lustre: server umount lustre-OST0001 complete [ 2730.464087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2730.466553] LustreError: Skipped 1 previous similar message [ 2730.468962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2734.309219] Lustre: server umount lustre-MDT0000 complete [ 2735.586480] LustreError: 193207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415614 with bad export cookie 11494688748912486612 [ 2735.592101] LustreError: 193207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2735.592896] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2735.727375] Lustre: server umount lustre-MDT0001 complete [ 2743.661063] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:00:22 (1784415622) [ 2745.746944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.841184] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2745.854078] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2745.923086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.925922] Lustre: Skipped 1 previous similar message [ 2747.142444] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2749.864826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.902665] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2749.914332] Lustre: 199480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2751.304139] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2753.892915] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2755.822439] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2758.099468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2758.167817] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2760.133260] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2762.529187] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2763.287408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 2764.938274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2764.963983] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2767.019387] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2769.396609] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2770.405441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.407408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2770.409786] Lustre: Skipped 2 previous similar messages [ 2770.411387] Lustre: Skipped 6 previous similar messages [ 2771.995852] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2776.322095] Lustre: server umount lustre-OST0000 complete [ 2777.063441] LustreError: 201327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.070253] LustreError: 201327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2787.296957] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2793.031576] Lustre: server umount lustre-OST0001 complete [ 2795.999768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2796.001700] LustreError: Skipped 1 previous similar message [ 2800.379212] Lustre: server umount lustre-MDT0000 complete [ 2801.638614] LustreError: 199472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415680 with bad export cookie 11494688748912487970 [ 2801.641197] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2801.642341] LustreError: 199472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2801.762030] Lustre: server umount lustre-MDT0001 complete [ 2809.444373] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:01:28 (1784415688) [ 2814.668604] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 2817.913984] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2829.754182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.846091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2829.856781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2829.897163] Lustre: lustre-MDT0000: new disk, initializing [ 2829.919677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2829.921518] Lustre: Skipped 3 previous similar messages [ 2829.926450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2831.217245] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2835.253615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.294960] Lustre: 207548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2835.331807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2835.333328] Lustre: Skipped 1 previous similar message [ 2835.366285] Lustre: lustre-MDT0001: new disk, initializing [ 2835.406683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2835.410024] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2836.709062] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2840.345186] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2842.232403] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2844.415853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2844.511521] Lustre: lustre-OST0000: new disk, initializing [ 2844.513236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2844.515450] Lustre: 209514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2846.411048] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2846.968570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2846.972724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2847.017051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2850.064509] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2851.364369] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2851.426734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2852.699466] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2852.770840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2857.440743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2857.442720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.443666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2857.445668] Lustre: Skipped 5 previous similar messages [ 2857.447222] Lustre: Skipped 10 previous similar messages [ 2859.541818] Lustre: server umount lustre-OST0000 complete [ 2875.359104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2875.415119] Lustre: server umount lustre-MDT0000 complete [ 2876.637541] LustreError: 208342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415755 with bad export cookie 11494688748912489713 [ 2876.641265] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2876.642041] LustreError: 208342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2876.763370] Lustre: server umount lustre-MDT0001 complete [ 2879.637824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.796630] LustreError: 211729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.801178] LustreError: 211729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2881.105285] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2884.009210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.599917] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2888.110807] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2890.015894] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2892.343076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2894.647384] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2897.224085] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2898.815260] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2906.010776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 2907.299032] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2907.361675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2910.175927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2910.178336] LustreError: Skipped 2 previous similar messages [ 2914.135970] Lustre: server umount lustre-OST0000 complete [ 2916.129134] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2916.207709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2917.531591] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2941.140785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.266124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2941.269877] Lustre: Skipped 5 previous similar messages [ 2943.240296] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2945.750022] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2946.251026] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2949.260044] Lustre: server umount lustre-OST0000 complete [ 2950.419092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.422232] Lustre: Skipped 5 previous similar messages [ 2952.529948] Lustre: server umount lustre-MDT0000 complete [ 2953.699984] LustreError: 211755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415832 with bad export cookie 11494688748912490581 [ 2953.702848] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2953.704779] LustreError: 211755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2953.824786] Lustre: server umount lustre-MDT0001 complete [ 2956.601720] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:03:55 (1784415835) [ 2958.864069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.027076] LustreError: 217752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.503208] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2963.315632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.791741] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2967.227434] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2969.059225] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2971.578044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2973.830399] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2976.179735] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2977.483139] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2984.671944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 2986.101976] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2986.190560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2988.411589] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2988.496233] Lustre: lustre-OST0001: new disk, initializing [ 2988.497968] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2988.500845] Lustre: 221167:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2990.543288] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2994.164913] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2995.618814] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2996.762357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2996.766599] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2996.804966] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2997.752119] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2999.155280] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2999.224720] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3001.828088] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3001.828302] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3001.831300] LustreError: Skipped 1 previous similar message [ 3001.834133] Lustre: Skipped 12 previous similar messages [ 3005.950382] Lustre: server umount lustre-OST0001 complete [ 3007.886676] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3007.958812] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3009.391149] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3009.478363] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3025.164347] LustreError: 219571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.170539] LustreError: 219571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3033.032257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3035.038142] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3037.481458] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3037.968384] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3038.182321] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3038.186262] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3038.186791] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3038.191600] Lustre: Skipped 1 previous similar message [ 3044.734661] Lustre: server umount lustre-OST0001 complete [ 3062.874033] Lustre: server umount lustre-OST0000 complete [ 3070.292450] Lustre: server umount lustre-MDT0000 complete [ 3071.497270] LustreError: 218485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784415950 with bad export cookie 11494688748912491512 [ 3071.500599] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3071.500764] LustreError: 218485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3077.673847] Lustre: server umount lustre-MDT0001 complete [ 3087.514884] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:06:06 (1784415966) [ 3090.172828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3090.270057] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3090.278871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3090.359787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3090.362069] Lustre: Skipped 5 previous similar messages [ 3092.084079] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3095.291167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.324903] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3095.340118] Lustre: 226416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3097.188715] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3099.790640] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3102.169959] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3105.039982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3105.162264] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3107.564600] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3110.589643] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3115.282145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3115.312273] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3117.882050] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3120.331855] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3120.852352] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3122.305711] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3122.390082] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3123.755558] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3123.826891] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3125.593497] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff959849387000.ost_server_uuid 50 [ 3126.054228] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff959849387000.ost_server_uuid in FULL state after 0 sec [ 3126.386659] Lustre: Permanently deactivating lustre-OST0001 [ 3126.393667] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3127.177372] Lustre: Permanently reactivating lustre-OST0001 [ 3131.362449] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3131.363876] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3131.365303] LustreError: Skipped 5 previous similar messages [ 3131.369040] Lustre: Skipped 8 previous similar messages [ 3134.208811] Lustre: server umount lustre-OST0001 complete [ 3141.579686] Lustre: server umount lustre-OST0000 complete [ 3148.958188] Lustre: server umount lustre-MDT0000 complete [ 3150.180335] LustreError: 227156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416029 with bad export cookie 11494688748912492786 [ 3150.184033] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3150.187254] LustreError: 227156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3150.310961] Lustre: server umount lustre-MDT0001 complete [ 3157.977761] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:07:17 (1784416037) [ 3160.368696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.475221] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3160.485311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3160.487966] Lustre: Skipped 5 previous similar messages [ 3161.871224] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3164.630687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.662576] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3164.673143] Lustre: 233096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3166.213479] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3168.537056] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3170.307951] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3172.451248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3172.540740] Lustre: Permanently deactivating lustre-OST0000 [ 3172.545542] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3173.743881] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3174.504094] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3177.465465] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3179.948400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3182.054652] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3182.685692] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3184.536550] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3185.084409] Lustre: Permanently reactivating lustre-OST0000 [ 3191.803635] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3191.806035] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3191.810343] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3191.836474] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3191.848199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3203.660613] Lustre: server umount lustre-OST0001 complete [ 3208.681566] LustreError: 234941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3208.689870] LustreError: 234941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 3211.674577] Lustre: server umount lustre-OST0000 complete [ 3219.365761] Lustre: server umount lustre-MDT0000 complete [ 3221.057305] LustreError: 236686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416100 with bad export cookie 11494688748912494424 [ 3221.063884] LustreError: 236686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3221.064046] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3221.220895] Lustre: server umount lustre-MDT0001 complete [ 3225.881273] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3227.391462] Key type lgssc unregistered [ 3227.553685] LNet: 237971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3227.558493] LNetError: 237971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3227.570529] LNet: Removed LNI 192.168.201.145@tcp [ 3228.044687] Key type .llcrypt unregistered [ 3228.047112] Key type ._llcrypt unregistered [ 3238.564486] Key type ._llcrypt registered [ 3238.565986] Key type .llcrypt registered [ 3238.631789] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3239.053985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3239.091253] alg: No test for adler32 (adler32-zlib) [ 3239.956912] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3240.054629] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3241.647191] Key type lgssc registered [ 3242.273615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3246.952680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3246.966048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3248.161975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3249.741125] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3253.777502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.840332] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3253.856088] Lustre: 239403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3253.863183] Lustre: Permanently deactivating lustre-MDT0001 [ 3253.871710] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3254.049565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3255.022546] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3255.024441] LustreError: 239446:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3255.027393] LustreError: 239446:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3255.030256] LustreError: 239446:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 3255.605467] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3258.925755] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3261.132308] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3263.733974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3263.882886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.933429] LustreError: 241255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.939274] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3264.943336] LustreError: 241255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3266.340535] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3269.536244] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3270.115847] LustreError: 241255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3272.435796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3272.539681] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3273.588296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3273.592278] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3273.593581] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3274.889437] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3278.234532] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3279.976685] Lustre: Permanently reactivating lustre-MDT0001 [ 3279.986665] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3279.989024] Lustre: Skipped 2 previous similar messages [ 3285.780656] Lustre: lustre-MDT0001: Client d9fd7a79-4293-42bc-a9e4-b2b6209d135c (at 192.168.201.45@tcp) reconnecting [ 3287.985101] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3288.501792] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3290.149581] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3290.220357] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3291.491547] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3291.560894] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3292.325666] Lustre: Permanently deactivating lustre-MDT0001 [ 3292.330840] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3292.332608] Lustre: Skipped 2 previous similar messages [ 3297.793272] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3304.416725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.419230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.420044] Lustre: Skipped 3 previous similar messages [ 3306.607538] Lustre: server umount lustre-MDT0000 complete [ 3308.295642] LustreError: 240155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416187 with bad export cookie 16215936014912946096 [ 3308.299933] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3308.303196] LustreError: 240155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3309.536280] LustreError: 239408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.545532] LustreError: 239408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3310.475390] Lustre: server umount lustre-MDT0001 complete [ 3312.165232] Lustre: server umount lustre-OST0000 complete [ 3317.164518] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 19:09:56 (1784416196) [ 3321.649546] Lustre: server umount lustre-OST0001 complete [ 3324.435865] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 3328.492891] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3342.764128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.878243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3342.883713] Lustre: Skipped 2 previous similar messages [ 3342.895132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3342.942983] Lustre: lustre-MDT0000: new disk, initializing [ 3342.974819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3342.982077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3344.553479] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3349.620790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3349.680956] Lustre: 247675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3349.710943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3349.713456] Lustre: Skipped 1 previous similar message [ 3349.750994] Lustre: lustre-MDT0001: new disk, initializing [ 3349.802227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3349.805590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3351.212461] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3355.317364] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3357.249751] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3360.306841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3360.477567] Lustre: lustre-OST0000: new disk, initializing [ 3360.480812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3360.485436] Lustre: 249644:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3360.534140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3360.537563] Lustre: Skipped 1 previous similar message [ 3363.102737] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3367.346290] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3367.934967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3367.941626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3367.975738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3373.521263] LustreError: 247681:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3378.270495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3378.347570] Lustre: lustre-OST0001: new disk, initializing [ 3378.349061] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3378.351224] Lustre: 250758:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3378.379373] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3380.623406] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3383.616094] LustreError: 247681:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3383.626754] LustreError: 248475:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3384.850568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3384.856228] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3384.899218] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3384.936353] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3387.708513] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3393.719129] LustreError: 248475:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3395.043883] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3395.046674] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.052065] Lustre: Skipped 1 previous similar message [ 3395.054214] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3395.056688] Lustre: Skipped 4 previous similar messages [ 3396.373814] Lustre: lustre-OST0001: Not available for connect from 192.168.201.45@tcp (stopping) [ 3400.170336] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3400.595706] Lustre: server umount lustre-OST0001 complete [ 3401.487381] LustreError: 249630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.408467] LustreError: 249629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.419790] LustreError: 249629:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3413.985278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3413.985425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.989811] LustreError: Skipped 2 previous similar messages [ 3413.997401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3414.004538] Lustre: Skipped 2 previous similar messages [ 3420.645751] LustreError: 249628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.655763] LustreError: 249628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3425.764311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3425.766144] Lustre: Skipped 5 previous similar messages [ 3427.295111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3427.345602] Lustre: server umount lustre-OST0000 complete [ 3430.882970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3430.892214] Lustre: Skipped 2 previous similar messages [ 3434.788913] Lustre: server umount lustre-MDT0000 complete [ 3436.190856] LustreError: 247668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416315 with bad export cookie 16215936014912948903 [ 3436.197111] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3436.202467] LustreError: 247668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3442.435088] Lustre: server umount lustre-MDT0001 complete [ 3446.650680] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3447.772851] Key type lgssc unregistered [ 3447.938529] LNet: 252791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3447.942792] LNetError: 252791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3447.952567] LNet: Removed LNI 192.168.201.145@tcp [ 3448.406220] Key type .llcrypt unregistered [ 3448.407604] Key type ._llcrypt unregistered [ 3462.680757] Key type ._llcrypt registered [ 3462.681758] Key type .llcrypt registered [ 3462.726606] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3463.207717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3463.217307] alg: No test for adler32 (adler32-zlib) [ 3464.076024] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3464.167122] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3465.751244] Key type lgssc registered [ 3466.133994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3468.666332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3468.671883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.756272] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3469.765431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3469.829827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.213425] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3474.098714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.135956] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3474.149122] Lustre: 254974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3474.253344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3475.561835] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3477.931674] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3479.698602] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3481.970392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3482.051123] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3482.121971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.116974] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3486.903283] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3487.287494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3533.284838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.285420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3533.287966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3533.290591] Lustre: Skipped 1 previous similar message [ 3533.293955] LustreError: Skipped 1 previous similar message [ 3536.824169] Lustre: server umount lustre-OST0000 complete [ 3538.595488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3543.121080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3543.269353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3545.451582] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3547.932277] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3553.508702] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3553.512083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3553.512153] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3553.516649] Lustre: Skipped 1 previous similar message [ 3584.480661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3584.482836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3584.484542] LustreError: Skipped 1 previous similar message [ 3584.489126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3584.491660] Lustre: Skipped 2 previous similar messages [ 3587.024339] Lustre: server umount lustre-OST0000 complete [ 3589.600612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.601363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3589.603737] Lustre: Skipped 2 previous similar messages [ 3594.720019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3594.721528] Lustre: Skipped 2 previous similar messages [ 3599.839898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3599.841427] Lustre: Skipped 1 previous similar message [ 3602.399100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3602.464272] Lustre: server umount lustre-MDT0000 complete [ 3603.664173] LustreError: 254965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416482 with bad export cookie 18386501108075857275 [ 3603.666668] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3603.667122] LustreError: 254965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3603.774792] Lustre: server umount lustre-MDT0001 complete [ 3607.885568] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3608.758433] Key type lgssc unregistered [ 3608.874381] LNet: 260386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3608.876498] LNetError: 260386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3608.884278] LNet: Removed LNI 192.168.201.145@tcp [ 3609.151915] Key type .llcrypt unregistered [ 3609.153047] Key type ._llcrypt unregistered [ 3620.196334] Key type ._llcrypt registered [ 3620.197329] Key type .llcrypt registered [ 3620.248471] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3620.598145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3620.633222] alg: No test for adler32 (adler32-zlib) [ 3621.550396] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3621.679953] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3623.279172] Key type lgssc registered [ 3623.805278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3626.961532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3626.970124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3628.144692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.442178] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3632.678820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3632.831944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3634.286125] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3637.136411] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3639.394475] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3642.105893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3642.235738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3644.305272] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3647.268559] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3653.545182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3668.963777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3668.964273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3668.964569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3668.966607] Lustre: Skipped 1 previous similar message [ 3668.968516] LustreError: Skipped 1 previous similar message [ 3671.629321] Lustre: server umount lustre-OST0000 complete [ 3674.081108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3674.081178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.088152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3674.089690] Lustre: Skipped 2 previous similar messages [ 3679.200248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3679.202493] Lustre: Skipped 1 previous similar message [ 3684.319977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3684.321958] Lustre: Skipped 1 previous similar message [ 3687.391096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3687.446175] Lustre: server umount lustre-MDT0000 complete [ 3688.700119] LustreError: 261761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416567 with bad export cookie 625522975481374927 [ 3688.703133] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3688.703305] LustreError: 261761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3688.809428] Lustre: server umount lustre-MDT0001 complete [ 3692.538590] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3693.403771] Key type lgssc unregistered [ 3693.511369] LNet: 266102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3693.513293] LNetError: 266102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3693.519257] LNet: Removed LNI 192.168.201.145@tcp [ 3693.781989] Key type .llcrypt unregistered [ 3693.782911] Key type ._llcrypt unregistered [ 3700.114059] Key type ._llcrypt registered [ 3700.114950] Key type .llcrypt registered [ 3700.147668] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3700.416259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3700.424451] alg: No test for adler32 (adler32-zlib) [ 3701.263762] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3701.341462] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3702.911137] Key type lgssc registered [ 3703.222499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3705.508586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3705.514883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.671085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3707.956238] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3710.607772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3710.734306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3712.040940] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3715.382286] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3717.905213] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3721.161735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3721.328114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3723.734735] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3727.224552] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3731.563089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3741.152955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3741.154050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3741.161169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3741.164602] Lustre: Skipped 1 previous similar message [ 3746.273136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3746.275141] Lustre: Skipped 1 previous similar message [ 3747.222359] Lustre: server umount lustre-OST0000 complete [ 3751.394552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.395551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3751.401410] Lustre: Skipped 2 previous similar messages [ 3755.131201] Lustre: server umount lustre-MDT0000 complete [ 3756.514367] LustreError: 267300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3756.523342] LustreError: 267300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3756.897663] LustreError: 267282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416636 with bad export cookie 6838377429982745757 [ 3756.902582] LustreError: 267282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3756.903193] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3757.048620] Lustre: server umount lustre-MDT0001 complete [ 3761.612306] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3762.919296] Key type lgssc unregistered [ 3763.067522] LNet: 270994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3763.070787] LNetError: 270994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3763.079456] LNet: Removed LNI 192.168.201.145@tcp [ 3763.508557] Key type .llcrypt unregistered [ 3763.510147] Key type ._llcrypt unregistered [ 3773.466477] Key type ._llcrypt registered [ 3773.467522] Key type .llcrypt registered [ 3773.504576] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3773.796471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3773.804181] alg: No test for adler32 (adler32-zlib) [ 3774.666097] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3774.759153] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3776.359175] Key type lgssc registered [ 3776.947235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3780.886571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3780.898505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3782.070791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3783.749860] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3787.448063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3787.575579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3789.044114] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3791.968473] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3794.181342] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3796.897917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3797.054386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.507251] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3802.705740] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3807.333241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3823.078175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3823.081390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.083330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3823.089596] Lustre: Skipped 1 previous similar message [ 3823.094281] Lustre: Skipped 1 previous similar message [ 3826.990766] Lustre: server umount lustre-OST0000 complete [ 3833.313828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.314963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3833.319619] Lustre: Skipped 1 previous similar message [ 3833.324456] Lustre: Skipped 1 previous similar message [ 3834.978917] Lustre: server umount lustre-MDT0000 complete [ 3836.504836] LustreError: 275672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416715 with bad export cookie 4644857576464650380 [ 3836.509491] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.509646] LustreError: 275672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3836.650100] Lustre: server umount lustre-MDT0001 complete [ 3840.806920] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3842.088494] Key type lgssc unregistered [ 3842.262748] LNet: 276710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3842.269976] LNetError: 276710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3842.284649] LNet: Removed LNI 192.168.201.145@tcp [ 3842.769202] Key type .llcrypt unregistered [ 3842.770698] Key type ._llcrypt unregistered [ 3850.120792] Key type ._llcrypt registered [ 3850.121881] Key type .llcrypt registered [ 3850.179513] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3850.598811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3850.647484] alg: No test for adler32 (adler32-zlib) [ 3851.563190] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3851.674800] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3853.271207] Key type lgssc registered [ 3853.864208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3857.271285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3857.279893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3858.494912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.206865] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3864.254868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3864.472696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3866.230339] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3869.466686] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3872.075079] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3875.027619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3875.181531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3877.797893] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3881.088947] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3883.376827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3897.825337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3897.828684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.834578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3899.876106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3899.879471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3899.884938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.344179] Lustre: server umount lustre-OST0000 complete [ 3904.993445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3904.994651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3904.996370] Lustre: Skipped 1 previous similar message [ 3904.998201] Lustre: Skipped 1 previous similar message [ 3910.112731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3910.114397] Lustre: Skipped 2 previous similar messages [ 3915.232698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3915.236099] Lustre: Skipped 1 previous similar message [ 3917.279089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3917.335479] Lustre: server umount lustre-MDT0000 complete [ 3918.510205] LustreError: 277905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416797 with bad export cookie 15097370728976709975 [ 3918.512993] LustreError: 277905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3918.513030] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3918.639820] Lustre: server umount lustre-MDT0001 complete [ 3922.241296] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3923.302259] Key type lgssc unregistered [ 3923.429540] LNet: 281655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3923.431495] LNetError: 281655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3923.440535] LNet: Removed LNI 192.168.201.145@tcp [ 3923.745793] Key type .llcrypt unregistered [ 3923.746847] Key type ._llcrypt unregistered [ 3927.355680] Key type ._llcrypt registered [ 3927.356705] Key type .llcrypt registered [ 3927.393877] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3932.840354] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 3939.418940] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3939.826775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3939.855108] alg: No test for adler32 (adler32-zlib) [ 3940.781211] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 3940.917169] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3942.527229] Key type lgssc registered [ 3943.230805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3958.257001] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3958.266373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3959.375638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3959.387286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3959.426895] Lustre: lustre-MDT0000: new disk, initializing [ 3959.453478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.464223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3961.222600] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3966.234535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3966.289301] Lustre: 285177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3966.318197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3966.321033] Lustre: Skipped 1 previous similar message [ 3966.366047] Lustre: lustre-MDT0001: new disk, initializing [ 3966.402231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3966.417800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3966.421648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3968.115306] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3972.003272] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3973.915482] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3976.860536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3977.001416] Lustre: lustre-OST0000: new disk, initializing [ 3977.004880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3977.010464] Lustre: 287145:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3977.051248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3978.679519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3978.682692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3978.717493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3979.148990] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3983.296332] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3984.666928] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3984.734971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3986.636526] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3986.707865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3988.962329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3988.967096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.968646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3988.975450] Lustre: Skipped 1 previous similar message [ 3988.981887] Lustre: Skipped 1 previous similar message [ 3993.619743] Lustre: server umount lustre-OST0000 complete [ 3997.151864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3997.153831] LustreError: Skipped 1 previous similar message [ 3997.155510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3997.160563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3999.201271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3999.202281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.332981] Lustre: server umount lustre-MDT0000 complete [ 4003.074357] LustreError: 285172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416882 with bad export cookie 14221503514198424915 [ 4003.079668] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4003.213989] Lustre: server umount lustre-MDT0001 complete [ 4006.810969] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:21:25 (1784416885) [ 4009.592594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4009.800507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4011.253061] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4014.795189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4014.940771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4016.734659] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4020.085493] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4022.755074] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4026.188824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4026.362262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4028.828651] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4031.646640] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4045.799984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4045.801335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.806123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4045.811284] Lustre: Skipped 1 previous similar message [ 4045.817697] Lustre: Skipped 2 previous similar messages [ 4047.537834] Lustre: server umount lustre-OST0000 complete [ 4049.360482] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4049.363908] LustreError: Skipped 1 previous similar message [ 4049.529257] Lustre: server umount lustre-MDT0000 complete [ 4050.912451] LustreError: 290281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4050.912615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4050.920295] LustreError: 290281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4050.925398] Lustre: Skipped 1 previous similar message [ 4051.049219] LustreError: 289577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784416930 with bad export cookie 14221503514198425804 [ 4051.051553] LustreError: 290290:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4051.052816] LustreError: 289577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4051.052985] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4057.252239] Lustre: server umount lustre-MDT0001 complete [ 4062.351324] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 4063.943795] Key type lgssc unregistered [ 4064.095726] LNet: 293104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4064.100474] LNetError: 293104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4064.108618] LNet: Removed LNI 192.168.201.145@tcp [ 4064.596370] Key type .llcrypt unregistered [ 4064.598402] Key type ._llcrypt unregistered [ 4076.871570] Key type ._llcrypt registered [ 4076.873069] Key type .llcrypt registered [ 4076.942995] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 4084.549367] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4084.996503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4085.038298] alg: No test for adler32 (adler32-zlib) [ 4085.917558] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 4086.033931] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 4087.623242] Key type lgssc registered [ 4088.314195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4109.963745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4109.973711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4111.094621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4111.106553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4111.168110] Lustre: lustre-MDT0000: new disk, initializing [ 4111.195344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4111.203779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4113.095422] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4118.744157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4118.789854] Lustre: 296899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4118.824537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4118.827586] Lustre: Skipped 1 previous similar message [ 4118.873796] Lustre: lustre-MDT0001: new disk, initializing [ 4118.915566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4118.929880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4118.936814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4120.856087] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4125.185642] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4127.083357] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4129.715505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4129.832205] Lustre: lustre-OST2710: new disk, initializing [ 4129.833814] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4129.836252] Lustre: 298870:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4129.865523] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4130.315388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4130.321139] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4130.359399] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4131.848450] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4136.419770] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4139.327497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4139.398942] Lustre: lustre-OST03e8: new disk, initializing [ 4139.400995] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4139.403992] Lustre: 299933:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4139.438789] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4140.062191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4140.068799] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4140.111896] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4142.018742] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4145.177062] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4148.185845] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4148.280112] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4149.802076] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4149.884594] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4151.547321] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4151.655152] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4153.507216] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4153.584948] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4154.848430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4154.853084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4154.861554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4155.361687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4155.363900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4155.365021] Lustre: Skipped 2 previous similar messages [ 4155.367798] Lustre: Skipped 2 previous similar messages [ 4160.481379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4160.889214] Lustre: server umount lustre-MDT0000 complete [ 4162.615355] LustreError: 296892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417041 with bad export cookie 8719624035807108510 [ 4162.620833] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4162.621983] LustreError: 296892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4162.806955] Lustre: server umount lustre-MDT0001 complete [ 4173.848991] Lustre: server umount lustre-OST2710 complete [ 4186.114918] Lustre: server umount lustre-OST03e8 complete [ 4194.964182] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 4199.334416] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4216.669315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.802274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4216.818384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4216.864491] Lustre: lustre-MDT0000: new disk, initializing [ 4216.898735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4216.908061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4218.687087] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4224.357970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4224.466179] Lustre: 305173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4224.652894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4226.523843] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4231.505628] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4234.077282] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4237.442810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4237.598180] Lustre: lustre-OST0000: new disk, initializing [ 4237.600371] Lustre: Skipped 1 previous similar message [ 4237.602875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4237.606040] Lustre: Skipped 2 previous similar messages [ 4237.609614] Lustre: 307142:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4237.662290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4237.666580] Lustre: Skipped 1 previous similar message [ 4238.591412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4238.595765] Lustre: Skipped 1 previous similar message [ 4238.598158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4238.626374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4240.036134] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4244.900391] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4247.029586] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4247.142837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4248.939728] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4249.024702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4254.176995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4254.178891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4254.179794] LustreError: Skipped 1 previous similar message [ 4254.184797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4254.188439] Lustre: Skipped 4 previous similar messages [ 4256.058209] Lustre: server umount lustre-OST0000 complete [ 4259.297746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.298411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4259.300958] Lustre: Skipped 2 previous similar messages [ 4259.305073] Lustre: Skipped 1 previous similar message [ 4264.140664] Lustre: server umount lustre-MDT0000 complete [ 4266.081704] LustreError: 305166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417145 with bad export cookie 8719624035807110183 [ 4266.089603] LustreError: 305166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4266.090091] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4266.271376] Lustre: server umount lustre-MDT0001 complete [ 4270.156320] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:25:49 (1784417149) [ 4270.883443] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4271.827078] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:25:50 (1784417150) [ 4273.163335] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4285.587920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4285.733658] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4285.746225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4285.830773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4287.792285] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4292.326752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4292.383422] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4292.402259] Lustre: 310995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4294.375387] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4311.333517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4311.458706] Lustre: Denying initial registration attempt from nid 192.168.201.145@tcp, specified as failover [ 4311.463891] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4311.468715] LustreError: 312446:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4311.472429] LustreError: 312446:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4311.475607] LustreError: 312446:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4311.517039] Lustre: server umount lustre-OST0000 complete [ 4311.519501] LustreError: 312446:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4317.153433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4317.155428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4317.158839] Lustre: Skipped 1 previous similar message [ 4317.166718] Lustre: Skipped 1 previous similar message [ 4320.807723] Lustre: server umount lustre-MDT0000 complete [ 4322.273750] LustreError: 311007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4322.283821] LustreError: 311007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4322.714931] LustreError: 311739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417201 with bad export cookie 8719624035807111065 [ 4322.718485] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4322.724342] LustreError: 311739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4322.862992] Lustre: server umount lustre-MDT0001 complete [ 4327.201915] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 4328.618663] Key type lgssc unregistered [ 4328.737344] LNet: 313570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4328.739325] LNetError: 313570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4328.748303] LNet: Removed LNI 192.168.201.145@tcp [ 4329.035146] Key type .llcrypt unregistered [ 4329.036248] Key type ._llcrypt unregistered [ 4333.136435] Key type ._llcrypt registered [ 4333.137775] Key type .llcrypt registered [ 4333.187734] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4333.651988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4333.683300] alg: No test for adler32 (adler32-zlib) [ 4334.551437] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 4334.643801] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 4336.231232] Key type lgssc registered [ 4336.822233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4365.752457] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4369.549407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4369.562602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4370.692775] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4370.702327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4370.793097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4372.846531] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4376.712615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4376.758119] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4376.773260] Lustre: 316307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4376.871650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4378.708664] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4382.280749] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4385.145707] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4401.124504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4401.223848] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4401.308445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4404.022607] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4407.759719] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4408.761821] Lustre: server umount lustre-OST0000 complete [ 4412.896362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4412.900610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4412.908633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4415.458645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4415.459596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4416.693872] Lustre: server umount lustre-MDT0000 complete [ 4418.593503] LustreError: 316300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417297 with bad export cookie 4617518285829896049 [ 4418.601448] LustreError: 316300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4418.603465] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4418.757200] Lustre: server umount lustre-MDT0001 complete [ 4423.433488] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 4425.077299] Key type lgssc unregistered [ 4425.266648] LNet: 319875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4425.272292] LNetError: 319875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4425.281789] LNet: Removed LNI 192.168.201.145@tcp [ 4425.774536] Key type .llcrypt unregistered [ 4425.776945] Key type ._llcrypt unregistered [ 4436.956447] Key type ._llcrypt registered [ 4436.957943] Key type .llcrypt registered [ 4437.017807] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4437.451221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4437.513069] alg: No test for adler32 (adler32-zlib) [ 4438.395913] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 4438.532572] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 4440.135200] Key type lgssc registered [ 4440.779348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4444.263940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4444.271257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4445.426599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4447.085504] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4450.595166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.724975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4452.247379] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4455.760792] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4457.715174] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4462.885110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4463.070296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4465.537318] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4468.596152] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4483.053593] LustreError: 323869:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200001b71:0x1:0x0].0x85ae478e (0000000046ce90c2) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4485.398846] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 4485.603667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4485.612385] Lustre: Skipped 2 previous similar messages [ 4489.208423] Lustre: server umount lustre-MDT0000 complete [ 4490.508336] LustreError: 323758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.517413] LustreError: 323758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4490.743462] LustreError: 321250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417369 with bad export cookie 18220348942930583247 [ 4490.748861] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4490.751236] LustreError: 321250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4495.629268] LustreError: 323758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4495.640894] LustreError: 323758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4495.644689] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.45@tcp (stopping) [ 4495.653028] Lustre: Skipped 3 previous similar messages [ 4495.841600] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4500.748622] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.45@tcp (stopping) [ 4500.748691] LustreError: 323758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.752122] Lustre: Skipped 1 previous similar message [ 4500.764326] LustreError: 323758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4505.055224] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4505.170962] Lustre: server umount lustre-MDT0001 complete [ 4508.358754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4511.449559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4516.319643] LustreError: 320579:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003c37355b x1871097172616832/t0(0) o250->MGC192.168.201.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4516.434049] LustreError: 323101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.441660] LustreError: 323101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4516.459627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4516.478198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4517.690489] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4520.437423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.571105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4521.834713] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4524.195710] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4527.076895] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4531.173238] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4531.850189] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 4533.668468] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4534.165109] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4536.289996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4536.291320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4536.293090] Lustre: Skipped 1 previous similar message [ 4536.293094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4536.293104] Lustre: Skipped 1 previous similar message [ 4541.409788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4541.412559] Lustre: Skipped 2 previous similar messages [ 4541.460284] Lustre: server umount lustre-OST0000 complete [ 4546.530758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4546.531263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4548.841093] Lustre: server umount lustre-MDT0000 complete [ 4550.145126] LustreError: 321248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417429 with bad export cookie 18220348942930590366 [ 4550.148786] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4550.149711] LustreError: 321248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4550.271470] Lustre: server umount lustre-MDT0001 complete [ 4554.474046] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 4555.875234] Key type lgssc unregistered [ 4556.021489] LNet: 327554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4556.024734] LNetError: 327554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4556.036572] LNet: Removed LNI 192.168.201.145@tcp [ 4556.450495] Key type .llcrypt unregistered [ 4556.451572] Key type ._llcrypt unregistered [ 4565.822599] Key type ._llcrypt registered [ 4565.824745] Key type .llcrypt registered [ 4565.865683] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4566.188649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4566.236235] alg: No test for adler32 (adler32-zlib) [ 4567.086212] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 4567.172843] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 4568.751234] Key type lgssc registered [ 4569.137735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4571.757805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4571.763787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4572.935297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4574.578935] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4578.272829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.490737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4579.948947] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4583.015540] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4585.262820] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4587.810566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4587.811630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4587.818530] Lustre: Skipped 1 previous similar message [ 4593.122624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4593.126715] Lustre: Skipped 2 previous similar messages [ 4598.241574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4598.246066] Lustre: Skipped 1 previous similar message [ 4601.311217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4601.412289] Lustre: server umount lustre-MDT0000 complete [ 4603.181322] LustreError: 328973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417482 with bad export cookie 6603975358772329929 [ 4603.188462] LustreError: 328973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4603.190172] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4603.333714] Lustre: server umount lustre-MDT0001 complete [ 4610.422903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.579415] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4610.595389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4610.697563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4612.711715] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4616.658744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4616.705355] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4616.717194] Lustre: 331533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4616.833793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4618.741659] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4622.006673] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4623.800881] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4629.029184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4629.141211] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4629.145147] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4629.148259] LustreError: 333412:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4629.151191] LustreError: 333412:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4629.154256] LustreError: 333412:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4629.181904] Lustre: server umount lustre-OST0000 complete [ 4629.183272] LustreError: 333412:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4635.236263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4635.365292] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4635.485637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4638.290700] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4641.855450] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4643.372965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4645.272334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4645.318997] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4645.364288] Lustre: lustre-OST0001: new disk, initializing [ 4645.366544] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4645.369845] Lustre: 334953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4645.408092] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4647.890968] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4651.274430] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4651.554159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4651.560430] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4651.619312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4656.612792] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4656.612916] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4656.617240] LustreError: Skipped 1 previous similar message [ 4656.625826] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4656.630102] Lustre: Skipped 1 previous similar message [ 4658.710351] Lustre: server umount lustre-OST0001 complete [ 4661.735554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4661.737751] LustreError: 333881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4661.739907] Lustre: Skipped 2 previous similar messages [ 4661.740586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4661.750229] LustreError: 333881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4661.751621] Lustre: Skipped 1 previous similar message [ 4666.584331] Lustre: server umount lustre-OST0000 complete [ 4668.383776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4668.386782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4671.969982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4671.972881] Lustre: Skipped 2 previous similar messages [ 4674.542486] Lustre: server umount lustre-MDT0000 complete [ 4676.352314] LustreError: 332270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417555 with bad export cookie 6603975358772330468 [ 4676.359321] LustreError: 332270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4676.360573] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4676.545441] Lustre: server umount lustre-MDT0001 complete [ 4681.379483] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 4682.730820] Key type lgssc unregistered [ 4682.903612] LNet: 336911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4682.908537] LNetError: 336911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4682.920631] LNet: Removed LNI 192.168.201.145@tcp [ 4683.336420] Key type .llcrypt unregistered [ 4683.337588] Key type ._llcrypt unregistered [ 4718.449781] Key type ._llcrypt registered [ 4718.451021] Key type .llcrypt registered [ 4718.501813] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 4725.725134] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4726.075273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4726.089247] alg: No test for adler32 (adler32-zlib) [ 4726.983660] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 4727.091768] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 4728.695209] Key type lgssc registered [ 4729.340417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4744.180068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4744.191307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4745.299585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4745.311398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4745.349159] Lustre: lustre-MDT0000: new disk, initializing [ 4745.375176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4745.382257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4747.133057] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4752.405607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4752.443735] Lustre: 341812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4752.470206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4752.471731] Lustre: Skipped 1 previous similar message [ 4752.510917] Lustre: lustre-MDT0001: new disk, initializing [ 4752.551774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4752.567898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4752.572441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4754.519887] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4759.713516] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4762.246326] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4765.277958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4765.399905] Lustre: lustre-OST0000: new disk, initializing [ 4765.401500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4765.403653] Lustre: 343781:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4765.434487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4767.246418] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4770.831880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4770.837111] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4770.891172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4771.076101] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4772.586204] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4772.699474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4774.701281] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4774.846868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4775.909626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4775.912110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4775.914429] LustreError: Skipped 1 previous similar message [ 4775.922680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4775.927410] Lustre: Skipped 1 previous similar message [ 4777.909730] Lustre: server umount lustre-OST0000 complete [ 4781.027736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4781.029387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4781.034627] Lustre: Skipped 2 previous similar messages [ 4781.038639] Lustre: Skipped 1 previous similar message [ 4786.145925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4786.150252] Lustre: Skipped 1 previous similar message [ 4791.265280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4791.268957] Lustre: Skipped 1 previous similar message [ 4794.335226] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4794.418245] Lustre: server umount lustre-MDT0000 complete [ 4796.268300] LustreError: 342610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417675 with bad export cookie 12684075116604523609 [ 4796.277601] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4796.471252] Lustre: server umount lustre-MDT0001 complete [ 4800.355371] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:34:39 (1784417679) [ 4803.915921] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:34:42 (1784417682) [ 4808.118534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4808.403748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4810.336563] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4813.792462] LustreError: 346534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4813.796887] LustreError: 346534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4813.837842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4813.967353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4815.418309] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4819.023617] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4821.924829] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4825.301437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4825.462893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4828.060205] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4831.237393] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4845.182696] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4845.782746] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4846.385939] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4847.189591] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4848.124979] Lustre: Failing over lustre-MDT0000 [ 4848.424793] Lustre: server umount lustre-MDT0000 complete [ 4850.144495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4850.149273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4850.157573] LustreError: 346534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4852.192452] LustreError: 346533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4852.193585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4852.196130] LustreError: 346533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4852.199211] Lustre: Skipped 1 previous similar message [ 4857.312410] LustreError: 346534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4857.316183] LustreError: 346534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4861.429275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.469376] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4861.558215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4861.582918] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4862.726340] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4863.242971] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4867.047543] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4867.056094] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4867.071296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4868.603849] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4869.049912] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4870.807948] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4871.639140] Lustre: Failing over lustre-MDT0000 [ 4871.795083] Lustre: server umount lustre-MDT0000 complete [ 4872.161447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4872.162918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4872.164144] LustreError: 346528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4872.164156] LustreError: 346528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4872.181075] Lustre: Skipped 1 previous similar message [ 4882.400987] LustreError: 346552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4882.411616] LustreError: 346552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4885.376613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4885.413143] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4885.522985] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4886.833821] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4888.352671] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4890.603417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4890.606381] Lustre: Skipped 2 previous similar messages [ 4890.619856] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4890.647605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4893.126577] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4893.846345] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4900.833750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4900.833915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4900.835759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4900.842037] Lustre: Skipped 2 previous similar messages [ 4902.705867] Lustre: server umount lustre-MDT0000 complete [ 4904.368261] LustreError: 346515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417783 with bad export cookie 12684075116604526262 [ 4904.371938] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4904.372615] LustreError: 346515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4904.493583] Lustre: server umount lustre-MDT0001 complete [ 4914.922254] Lustre: server umount lustre-OST0000 complete [ 4922.004164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.239474] LustreError: 352882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.251181] LustreError: 352882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4922.270371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4922.273285] Lustre: Skipped 1 previous similar message [ 4924.109443] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4928.006199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4929.868918] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4933.255659] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4935.251485] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4938.397498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4941.336541] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4944.834173] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4946.800690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4957.152538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4957.157052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4957.164111] Lustre: Skipped 1 previous similar message [ 4957.167748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4957.172757] Lustre: Skipped 2 previous similar messages [ 4963.115698] Lustre: server umount lustre-OST0000 complete [ 4973.537054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4973.540784] Lustre: Skipped 5 previous similar messages [ 4979.167126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4979.249145] Lustre: server umount lustre-MDT0000 complete [ 4980.989154] LustreError: 352862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417860 with bad export cookie 12684075116604527011 [ 4980.992051] LustreError: 352862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4980.992064] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4981.106657] Lustre: server umount lustre-MDT0001 complete [ 4985.454145] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 4986.891310] Key type lgssc unregistered [ 4987.065574] LNet: 356432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4987.069549] LNetError: 356432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4987.079444] LNet: Removed LNI 192.168.201.145@tcp [ 4987.464961] Key type .llcrypt unregistered [ 4987.466926] Key type ._llcrypt unregistered [ 4999.830493] Key type ._llcrypt registered [ 4999.831743] Key type .llcrypt registered [ 4999.883222] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5000.195195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5000.209811] alg: No test for adler32 (adler32-zlib) [ 5001.059726] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 5001.153135] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5002.727166] Key type lgssc registered [ 5003.266872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5006.950194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5006.962771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5008.145742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5009.934946] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5013.952543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5014.089092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5015.706855] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5019.076385] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5021.612481] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5024.862282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5025.018677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5027.422421] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5030.313859] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5033.192293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5049.827614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5049.827694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5049.832510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5049.835712] Lustre: Skipped 2 previous similar messages [ 5051.065589] Lustre: server umount lustre-MDT0000 complete [ 5052.705951] LustreError: 357907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417931 with bad export cookie 5152694978160547173 [ 5052.712429] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5052.712819] LustreError: 357907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5052.844946] Lustre: server umount lustre-MDT0001 complete [ 5064.557555] Lustre: server umount lustre-OST0000 complete [ 5070.781353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.965850] LustreError: 361654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5070.974116] LustreError: 361654:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5071.001409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5072.735831] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5076.449409] LustreError: 361654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5076.778468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5076.948764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5078.692874] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5081.758195] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5084.321928] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5087.505911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5087.689811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5090.159567] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5093.472707] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5095.925129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5107.172167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5107.177088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5107.179189] Lustre: Skipped 2 previous similar messages [ 5107.183229] Lustre: Skipped 3 previous similar messages [ 5112.121466] Lustre: server umount lustre-MDT0000 complete [ 5112.291091] LustreError: 361654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5112.300282] LustreError: 361654:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5113.834229] LustreError: 361636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784417993 with bad export cookie 5152694978160548272 [ 5113.837466] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5113.840794] LustreError: 361636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5113.990461] Lustre: server umount lustre-MDT0001 complete [ 5124.857590] Lustre: server umount lustre-OST0000 complete [ 5131.212258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5131.376564] LustreError: 365564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5131.398551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5133.026690] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5136.849938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5136.864675] LustreError: 365565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5138.197134] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5141.079245] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5143.383682] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5146.508798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5149.192615] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5152.617434] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5155.948219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5161.445790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5161.446055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5161.463648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5161.469525] Lustre: Skipped 1 previous similar message [ 5165.531144] Lustre: server umount lustre-OST0000 complete [ 5167.584324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5167.589810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5167.597875] Lustre: Skipped 1 previous similar message [ 5167.601572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5171.680343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5171.680915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5173.161077] Lustre: server umount lustre-MDT0000 complete [ 5174.543081] LustreError: 365546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418053 with bad export cookie 5152694978160549280 [ 5174.545454] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5174.548245] LustreError: 365546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5174.652267] Lustre: server umount lustre-MDT0001 complete [ 5178.574601] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5180.075140] Key type lgssc unregistered [ 5180.243649] LNet: 369095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5180.248213] LNetError: 369095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5180.256591] LNet: Removed LNI 192.168.201.145@tcp [ 5180.724104] Key type .llcrypt unregistered [ 5180.728087] Key type ._llcrypt unregistered [ 5192.657402] Key type ._llcrypt registered [ 5192.658754] Key type .llcrypt registered [ 5192.717133] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5193.064649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5193.094063] alg: No test for adler32 (adler32-zlib) [ 5193.946445] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 5194.045102] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5195.647174] Key type lgssc registered [ 5196.189459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5199.753526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5199.764467] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.770055] LustreError: 370512:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5199.780532] LustreError: 370512:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5199.784110] LustreError: 370512:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5199.787809] LustreError: 370512:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5199.791908] LustreError: 370512:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5203.011214] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5203.016299] LustreError: 370909:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5203.027555] LustreError: 370909:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5203.031987] LustreError: 370909:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5203.036442] LustreError: 370909:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5203.041439] LustreError: 370909:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5211.290830] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5212.407928] Key type lgssc unregistered [ 5212.523284] LNet: 371910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5212.525229] LNetError: 371910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5212.537247] LNet: Removed LNI 192.168.201.145@tcp [ 5212.856285] Key type .llcrypt unregistered [ 5212.857328] Key type ._llcrypt unregistered [ 5220.099217] Key type ._llcrypt registered [ 5220.100290] Key type .llcrypt registered [ 5220.139854] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 5225.722055] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5226.005640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5226.033028] alg: No test for adler32 (adler32-zlib) [ 5226.870352] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 5226.947529] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5228.519101] Key type lgssc registered [ 5228.821654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5238.780207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5238.784958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5239.859145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5239.867554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5239.929031] Lustre: lustre-MDT0000: new disk, initializing [ 5239.950575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5239.955663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5241.402629] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5245.474961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5245.509672] Lustre: 375038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5245.530807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5245.532230] Lustre: Skipped 1 previous similar message [ 5245.564907] Lustre: lustre-MDT0001: new disk, initializing [ 5245.592254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5245.602259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5245.605101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5246.717465] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5250.029527] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5251.589040] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5253.643834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5253.739055] Lustre: lustre-OST0000: new disk, initializing [ 5253.740508] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5253.742627] Lustre: 377005:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5253.765915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5255.464867] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5258.742358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5258.744993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5258.780376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5258.880255] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5260.049695] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5260.107190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5261.242336] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5261.300491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5263.842309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5263.842879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5263.843925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5263.845260] Lustre: Skipped 1 previous similar message [ 5267.984023] Lustre: server umount lustre-OST0000 complete [ 5271.520533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5271.524920] LustreError: Skipped 1 previous similar message [ 5271.527443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5271.535119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5271.538885] Lustre: Skipped 1 previous similar message [ 5274.080954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5274.082136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.250604] Lustre: server umount lustre-MDT0000 complete [ 5276.772845] LustreError: 375031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418156 with bad export cookie 11462073757678136996 [ 5276.779529] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5276.779750] LustreError: 375031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5276.926575] Lustre: server umount lustre-MDT0001 complete [ 5280.437902] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5281.090644] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:42:40 (1784418160) [ 5284.318060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5284.581517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5286.370630] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5289.645821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5289.802850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5291.484765] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5294.460163] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5296.636626] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5299.990027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5300.068057] LustreError: 381279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5300.076811] LustreError: 381279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5300.161815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5302.614882] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5306.075558] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5308.505608] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5308.585654] Lustre: lustre-OST0001: new disk, initializing [ 5308.588776] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5308.593121] Lustre: 382293:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5308.735560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5308.738689] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5308.769969] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5311.161851] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5316.103232] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5319.137121] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5319.137250] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5319.142046] LustreError: Skipped 1 previous similar message [ 5319.149662] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5319.155031] Lustre: Skipped 2 previous similar messages [ 5324.169725] Lustre: server umount lustre-OST0001 complete [ 5325.284510] LustreError: 381279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5325.288647] LustreError: 381279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5330.405864] LustreError: 381281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5330.414856] LustreError: 381281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5335.526068] LustreError: 381280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5335.535205] LustreError: 381280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5340.643861] LustreError: 382332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5340.646480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5340.649099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5340.649107] Lustre: Skipped 3 previous similar messages [ 5340.653251] LustreError: 382332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5340.680208] Lustre: Skipped 2 previous similar messages [ 5340.684310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5350.883447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5350.886377] LustreError: 381280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5350.887083] Lustre: Skipped 4 previous similar messages [ 5350.896100] LustreError: 381280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5351.391204] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5351.478974] Lustre: server umount lustre-OST0000 complete [ 5356.000807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5356.003963] Lustre: Skipped 1 previous similar message [ 5359.469536] Lustre: server umount lustre-MDT0000 complete [ 5361.255798] LustreError: 379485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418240 with bad export cookie 11462073757678137864 [ 5361.261889] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5361.265815] LustreError: 379485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5367.464749] Lustre: server umount lustre-MDT0001 complete [ 5370.527571] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5371.811453] Key type lgssc unregistered [ 5371.986514] LNet: 384277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5371.991077] LNetError: 384277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5372.001518] LNet: Removed LNI 192.168.201.145@tcp [ 5372.475660] Key type .llcrypt unregistered [ 5372.477848] Key type ._llcrypt unregistered [ 5393.873238] Key type ._llcrypt registered [ 5393.874334] Key type .llcrypt registered [ 5393.925252] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5394.471788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5394.498597] alg: No test for adler32 (adler32-zlib) [ 5395.396248] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 5395.503746] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5397.111181] Key type lgssc registered [ 5397.564574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5400.291818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5400.298388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5401.394062] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5401.403148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5401.499857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5403.539936] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5407.534926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.579564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5407.590339] Lustre: 387049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5407.697493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5409.239812] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5412.084995] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5413.946542] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5417.952813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5417.957703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5417.966075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5420.001969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5420.003407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5425.121860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5425.125522] Lustre: Skipped 2 previous similar messages [ 5429.215167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5429.294234] Lustre: server umount lustre-MDT0000 complete [ 5431.041143] LustreError: 387794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418310 with bad export cookie 8626277428955597488 [ 5431.046761] LustreError: 387794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5431.047598] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5431.196696] Lustre: server umount lustre-MDT0001 complete [ 5435.397164] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:45:14 (1784418314) [ 5438.214429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5438.452517] LustreError: 389565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.461953] LustreError: 389565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5438.481163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5440.007300] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5443.105229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.239327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5444.646601] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5447.807658] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5450.428050] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5453.016885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5453.096944] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5453.167312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5454.795418] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5458.059472] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5462.376720] Lustre: Permanently deactivating lustre-OST0000 [ 5462.386214] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5463.483049] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5464.555210] LustreError: 392406:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5464.560566] LustreError: 392406:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5464.564235] LustreError: 392406:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5465.192370] LustreError: 392454:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5465.196048] LustreError: 392454:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5471.906449] Lustre: server umount lustre-OST0000 complete [ 5473.167920] LustreError: 392703:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5473.171385] LustreError: 392703:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5474.272305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5474.276668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5474.283783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5476.322229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5481.440834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5481.444536] Lustre: Skipped 3 previous similar messages [ 5484.548527] Lustre: server umount lustre-MDT0000 complete [ 5486.369301] LustreError: 391398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418365 with bad export cookie 8626277428955598055 [ 5486.375064] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5486.376683] LustreError: 391398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5495.852985] Lustre: server umount lustre-MDT0001 complete [ 5499.021677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5500.725483] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5501.924179] LustreError: 393868:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5504.903852] Lustre: 394207:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.145@tcp [ 5505.889562] Lustre: server umount MGS complete [ 5510.263943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5510.401806] LustreError: 394931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5510.406229] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5510.407877] Lustre: Skipped 1 previous similar message [ 5510.425270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.848261] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5515.069424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.190097] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5516.712455] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5519.465604] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5521.304251] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5522.089254] Lustre: Permanently reactivating lustre-OST0000 [ 5522.096851] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5522.099586] Lustre: Skipped 2 previous similar messages [ 5546.975204] LustreError: 396678:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5550.502625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5550.678840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5550.682470] Lustre: Skipped 1 previous similar message [ 5553.291583] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5556.586585] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5566.434323] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5566.441072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5566.452087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5569.380501] Lustre: server umount lustre-OST0000 complete [ 5571.551786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5571.553810] LustreError: Skipped 1 previous similar message [ 5582.817728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5582.823392] Lustre: Skipped 8 previous similar messages [ 5585.375182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5585.456832] Lustre: server umount lustre-MDT0000 complete [ 5587.198638] LustreError: 394912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418466 with bad export cookie 8626277428955599077 [ 5587.203285] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5587.205215] LustreError: 394912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5587.362100] Lustre: server umount lustre-MDT0001 complete [ 5591.512735] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5592.736303] Key type lgssc unregistered [ 5592.865298] LNet: 399485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5592.867593] LNetError: 399485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5592.873444] LNet: Removed LNI 192.168.201.145@tcp [ 5593.197776] Key type .llcrypt unregistered [ 5593.198915] Key type ._llcrypt unregistered [ 5600.649746] Key type ._llcrypt registered [ 5600.651883] Key type .llcrypt registered [ 5600.710807] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 5608.089603] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5608.418476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5608.454845] alg: No test for adler32 (adler32-zlib) [ 5609.348143] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 5609.450350] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5611.063202] Key type lgssc registered [ 5611.664519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5628.548089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5628.560642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5629.709823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5629.729998] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5629.778562] Lustre: lustre-MDT0000: new disk, initializing [ 5629.819242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5629.831198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5631.621632] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5637.357153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5637.431660] Lustre: 402613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5637.478658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5637.482243] Lustre: Skipped 1 previous similar message [ 5637.528514] Lustre: lustre-MDT0001: new disk, initializing [ 5637.574985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5637.593194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5637.597879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5639.385395] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5644.390405] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5646.974528] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5650.322145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5650.510591] Lustre: lustre-OST0000: new disk, initializing [ 5650.513985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5650.518910] Lustre: 404584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5650.576871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5653.322395] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5657.998574] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5659.910263] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5660.681199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5660.688712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5660.742878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5661.063459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5662.862442] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5662.967210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5665.761485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5665.766759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5665.767224] LustreError: Skipped 1 previous similar message [ 5665.775731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5665.780407] Lustre: Skipped 1 previous similar message [ 5669.995560] Lustre: server umount lustre-OST0000 complete [ 5673.440625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5673.445097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.451996] Lustre: Skipped 1 previous similar message [ 5673.455281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5676.000918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5676.002164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5677.752818] Lustre: server umount lustre-MDT0000 complete [ 5679.594606] LustreError: 402608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418558 with bad export cookie 11494639530317063334 [ 5679.599542] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5679.601370] LustreError: 402608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5679.784156] Lustre: server umount lustre-MDT0001 complete [ 5683.754729] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5684.507453] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:49:23 (1784418563) [ 5687.690995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5687.944242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.640227] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5693.410820] LustreError: 407047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.418032] LustreError: 407047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5693.779581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5693.953631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5695.895875] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5699.067152] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5701.547224] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5704.510795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5704.689922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5707.051921] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5710.573658] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5711.488339] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5718.346026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5728.224752] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5728.229775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.238406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5728.242316] Lustre: Skipped 1 previous similar message [ 5731.426819] Lustre: server umount lustre-OST0000 complete [ 5734.369765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.371658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5734.384420] Lustre: Skipped 2 previous similar messages [ 5734.387961] Lustre: Skipped 2 previous similar messages [ 5739.245794] Lustre: server umount lustre-MDT0000 complete [ 5739.488901] LustreError: 407065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5739.498068] LustreError: 407065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5741.026313] LustreError: 407027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418620 with bad export cookie 11494639530317064202 [ 5741.031903] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5741.033204] LustreError: 407027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5741.186909] Lustre: server umount lustre-MDT0001 complete [ 5745.825802] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5747.401106] Key type lgssc unregistered [ 5747.572622] LNet: 410792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5747.577308] LNetError: 410792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5747.591746] LNet: Removed LNI 192.168.201.145@tcp [ 5748.096508] Key type .llcrypt unregistered [ 5748.098502] Key type ._llcrypt unregistered [ 5758.857204] Key type ._llcrypt registered [ 5758.858203] Key type .llcrypt registered [ 5758.903796] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5759.063282] Key type .llcrypt unregistered [ 5759.065044] Key type ._llcrypt unregistered [ 5768.466207] Key type ._llcrypt registered [ 5768.467837] Key type .llcrypt registered [ 5768.530452] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5768.873308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5768.879750] alg: No test for adler32 (adler32-zlib) [ 5769.763432] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 5769.883600] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5771.487221] Key type lgssc registered [ 5772.048061] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5776.073791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5776.084553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5777.300794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.128408] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5783.368363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5783.566364] LustreError: 412976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5783.572847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5783.576263] LustreError: 412976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5786.316625] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5787.681901] LustreError: 412981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.695366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5787.696847] LustreError: 412981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5789.880228] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5792.743041] LustreError: 412980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5792.752184] LustreError: 412980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5793.291040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5793.502677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5795.547523] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5804.010805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.011769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5804.014917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5804.017976] Lustre: Skipped 1 previous similar message [ 5807.011459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5807.015907] Lustre: Skipped 1 previous similar message [ 5809.522219] Lustre: server umount lustre-OST0000 complete [ 5812.196179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.198077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5812.203836] Lustre: Skipped 1 previous similar message [ 5812.211190] Lustre: Skipped 1 previous similar message [ 5817.313599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5817.317145] Lustre: Skipped 1 previous similar message [ 5817.597975] Lustre: server umount lustre-MDT0000 complete [ 5819.413544] LustreError: 415089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418698 with bad export cookie 15677848560852302415 [ 5819.419416] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5819.420270] LustreError: 415089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5819.573544] Lustre: server umount lustre-MDT0001 complete [ 5824.450722] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5826.057374] Key type lgssc unregistered [ 5826.226778] LNet: 416130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5826.231473] LNetError: 416130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5826.241806] LNet: Removed LNI 192.168.201.145@tcp [ 5826.766313] Key type .llcrypt unregistered [ 5826.769536] Key type ._llcrypt unregistered [ 5839.309729] Key type ._llcrypt registered [ 5839.311375] Key type .llcrypt registered [ 5839.378858] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5839.994290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5840.001462] alg: No test for adler32 (adler32-zlib) [ 5840.883656] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 5840.987455] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5842.591148] Key type lgssc registered [ 5843.247622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5847.265103] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5847.276633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5862.751401] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5867.871278] LustreError: 417497:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.145@tcp: failed processing log, type 4: rc = -110 [ 5894.495311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.294428] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5900.150118] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5903.414534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5903.672452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5903.703477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5905.403704] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5908.771758] LustreError: 417523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5908.780773] LustreError: 417523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5909.461041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5909.668424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5911.400721] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5914.969659] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5917.627294] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5924.320272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5924.322773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.327619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5925.346761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5925.351870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5925.356387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5926.626188] Lustre: server umount lustre-OST0000 complete [ 5930.468090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5930.469041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5930.481973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5930.487108] Lustre: Skipped 2 previous similar messages [ 5934.577834] Lustre: server umount lustre-MDT0000 complete [ 5935.585090] LustreError: 418538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5935.594502] LustreError: 418538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5936.370934] LustreError: 417504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418815 with bad export cookie 13182130434076500688 [ 5936.379809] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5936.380872] LustreError: 417504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5936.555319] Lustre: server umount lustre-MDT0001 complete [ 5941.875304] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5943.452296] Key type lgssc unregistered [ 5943.604856] LNet: 421063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5943.609797] LNetError: 421063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5943.618648] LNet: Removed LNI 192.168.201.145@tcp [ 5944.096750] Key type .llcrypt unregistered [ 5944.099787] Key type ._llcrypt unregistered [ 5956.875362] Key type ._llcrypt registered [ 5956.877088] Key type .llcrypt registered [ 5956.928676] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5957.361854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5957.406912] alg: No test for adler32 (adler32-zlib) [ 5958.283885] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 5958.368268] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5959.943169] Key type lgssc registered [ 5960.517272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5964.281837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5964.292322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.445629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5967.211375] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5971.285914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5971.465810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5973.239727] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5976.694098] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5979.427121] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5982.512409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5982.627608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5984.769852] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5988.252126] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5989.803805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5997.024399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5997.029093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5997.036974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6000.923037] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 6001.124338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6002.929581] Lustre: server umount lustre-MDT0000 complete [ 6004.035166] LustreError: 422439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.45@tcp arrived at 1784418883 with bad export cookie 1053177735557398563 [ 6004.974991] LustreError: 422437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418884 with bad export cookie 1053177735557397947 [ 6004.982215] LustreError: 422437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6004.982947] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6006.241214] LustreError: 422458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6006.241487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6006.250164] LustreError: 422458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6006.254705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.266285] Lustre: Skipped 1 previous similar message [ 6006.269773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6006.273211] Lustre: Skipped 3 previous similar messages [ 6009.248362] LustreError: 422457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6009.249096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6009.257405] LustreError: 422457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6011.109624] Lustre: server umount lustre-OST0000 complete [ 6014.372917] Lustre: server umount lustre-MDT0001 complete [ 6019.410559] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 6020.886256] Key type lgssc unregistered [ 6021.052753] LNet: 426139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6021.057451] LNetError: 426139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6021.072542] LNet: Removed LNI 192.168.201.145@tcp [ 6021.542668] Key type .llcrypt unregistered [ 6021.544787] Key type ._llcrypt unregistered [ 6033.824470] Key type ._llcrypt registered [ 6033.825784] Key type .llcrypt registered [ 6033.863809] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6034.194084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6034.201185] alg: No test for adler32 (adler32-zlib) [ 6035.062396] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 6035.153900] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 6036.727170] Key type lgssc registered [ 6037.234713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6040.970923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6040.981407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6042.164565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6043.958282] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6048.005247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6048.175829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6049.859267] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6053.507866] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6056.264309] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6059.577962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6059.750098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6061.922296] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6065.351702] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6066.924606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6073.077219] LustreError: 428280:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 6073.083742] LustreError: 428280:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 6077.197259] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.45@tcp (stopping) [ 6077.920314] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6077.924657] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6077.932489] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6078.437120] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.263960] Lustre: server umount lustre-MDT0001 complete [ 6083.554444] LustreError: 427567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6083.563720] LustreError: 427567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6088.673115] LustreError: 427549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6088.682371] LustreError: 427549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6093.794664] LustreError: 427548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6093.803735] LustreError: 427548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6098.914680] LustreError: 427549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6098.923562] LustreError: 427549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6104.033609] LustreError: 427567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.042706] LustreError: 427567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6108.128281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6108.132421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6108.140271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6108.143754] Lustre: Skipped 2 previous similar messages [ 6111.974225] Lustre: server umount lustre-OST0000 complete [ 6113.919492] Lustre: server umount lustre-MDT0000 complete [ 6120.140487] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 6121.561214] Key type lgssc unregistered [ 6121.726561] LNet: 431257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6121.732507] LNetError: 431257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6121.744546] LNet: Removed LNI 192.168.201.145@tcp [ 6122.202718] Key type .llcrypt unregistered [ 6122.204834] Key type ._llcrypt unregistered [ 6133.471487] Key type ._llcrypt registered [ 6133.472815] Key type .llcrypt registered [ 6133.529874] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 6140.929295] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6141.255153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6141.294411] alg: No test for adler32 (adler32-zlib) [ 6142.143545] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 6142.237143] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 6143.815138] Key type lgssc registered [ 6144.379698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6162.600376] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6169.786441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6169.795956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6170.899114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6170.915348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6170.960700] Lustre: lustre-MDT0000: new disk, initializing [ 6170.983635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6170.993306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6172.329334] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6179.636594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6179.790934] Lustre: lustre-OST0000: new disk, initializing [ 6179.793934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6179.797121] Lustre: Skipped 1 previous similar message [ 6179.800226] Lustre: 436263:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6179.835077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6182.369977] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6186.506109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6186.510848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6186.522981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6187.162679] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6194.674155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6194.745790] Lustre: 435245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6194.799865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6194.852309] Lustre: lustre-MDT0001: new disk, initializing [ 6194.913221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6194.933896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6194.940884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6196.687643] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6200.743279] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6200.811793] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6202.136257] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6202.210692] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6206.745739] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.45@tcp (stopping) [ 6206.949528] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6206.949666] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6211.853563] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.45@tcp (stopping) [ 6211.858656] Lustre: Skipped 2 previous similar messages [ 6216.972942] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.45@tcp (stopping) [ 6216.977890] Lustre: Skipped 2 previous similar messages [ 6218.207188] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6218.290993] Lustre: server umount lustre-MDT0001 complete [ 6227.428679] LustreError: 435254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6227.437618] LustreError: 435254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6232.546212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6232.547571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6232.547866] LustreError: 435255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6232.568220] Lustre: Skipped 1 previous similar message [ 6232.571924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6232.575952] Lustre: Skipped 2 previous similar messages [ 6237.416044] Lustre: server umount lustre-OST0000 complete [ 6237.667344] LustreError: 435254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6245.523082] Lustre: server umount lustre-MDT0000 complete [ 6251.424693] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 6252.911968] Key type lgssc unregistered [ 6253.083551] LNet: 440009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6253.088739] LNetError: 440009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6253.100598] LNet: Removed LNI 192.168.201.145@tcp [ 6253.572722] Key type .llcrypt unregistered [ 6253.574859] Key type ._llcrypt unregistered [ 6271.454825] Key type ._llcrypt registered [ 6271.455942] Key type .llcrypt registered [ 6271.500319] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 6271.657350] Key type .llcrypt unregistered [ 6271.659383] Key type ._llcrypt unregistered [ 6280.282482] Key type ._llcrypt registered [ 6280.284262] Key type .llcrypt registered [ 6280.348732] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6280.667660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6280.677402] alg: No test for adler32 (adler32-zlib) [ 6281.550986] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 6281.656879] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 6283.255087] Key type lgssc registered [ 6283.652371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6294.868668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6294.878351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6295.990141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6296.005413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6296.053935] Lustre: lustre-MDT0000: new disk, initializing [ 6296.079544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6296.086370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6297.787652] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6303.311621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6303.384735] Lustre: 443361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6303.428545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6303.432030] Lustre: Skipped 1 previous similar message [ 6303.483323] Lustre: lustre-MDT0001: new disk, initializing [ 6303.523357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6303.543939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6303.551426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6305.198609] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6310.232838] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6312.788830] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6316.083043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6316.244499] Lustre: lustre-OST0000: new disk, initializing [ 6316.247204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6316.251954] Lustre: 445328:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6316.304688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6318.961732] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6323.729506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6323.735701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6323.794238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6323.879622] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6328.804938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6328.806530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6328.811954] Lustre: Skipped 1 previous similar message [ 6332.852385] Lustre: server umount lustre-MDT0000 complete [ 6334.609271] LustreError: 443352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784419213 with bad export cookie 5411828683902657192 [ 6334.614472] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6334.615909] LustreError: 443352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6334.772314] Lustre: server umount lustre-MDT0001 complete [ 6346.751076] Lustre: server umount lustre-OST0000 complete [ 6351.930600] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:00:30 (1784419230) [ 6355.648568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6355.835442] LustreError: 447354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.843509] LustreError: 447354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6355.872563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6357.762948] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6361.057695] LustreError: 447355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6361.812673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6361.978315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6363.670362] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6367.117840] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6369.808582] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6386.084615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6386.191664] Lustre: Found index 0 for lustre-OST0000, updating log [ 6386.286683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6388.949247] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6392.491429] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6402.667391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6408.162337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6408.166395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6408.168160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6408.173950] Lustre: Skipped 2 previous similar messages [ 6408.177441] Lustre: Skipped 2 previous similar messages [ 6409.249903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6411.110510] Lustre: server umount lustre-OST0000 complete [ 6425.570191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6425.571130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6425.577206] Lustre: Skipped 1 previous similar message [ 6425.583032] Lustre: Skipped 2 previous similar messages [ 6430.508982] Lustre: server umount lustre-MDT0000 complete [ 6430.688812] LustreError: 447355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6430.698132] LustreError: 447355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6432.274475] LustreError: 447334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784419311 with bad export cookie 5411828683902658585 [ 6432.280418] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6432.281779] LustreError: 447334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6432.425247] Lustre: server umount lustre-MDT0001 complete [ 6436.690624] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:01:55 (1784419315) [ 6455.775321] LNet: 1 local NIs in recovery (showing 1): 192.168.201.145@tcp [ 7023.413714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7023.599567] LustreError: 453789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7023.629249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7025.349337] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7028.706228] LustreError: 453790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7029.318687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7029.454501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7031.154560] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7034.281752] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7036.847677] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7053.092829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7053.230623] Lustre: Found index 0 for lustre-OST0000, updating log [ 7053.344109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7055.931150] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7059.227184] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7061.474141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7061.478755] LustreError: Skipped 1 previous similar message [ 7061.481360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7061.489515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7062.242703] Lustre: server umount lustre-OST0000 complete [ 7064.015218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7064.017201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7064.022320] Lustre: Skipped 1 previous similar message [ 7067.040985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7067.044541] Lustre: Skipped 2 previous similar messages [ 7072.225092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7072.228765] Lustre: Skipped 1 previous similar message [ 7077.345553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7077.349543] Lustre: Skipped 1 previous similar message [ 7078.367109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7078.444774] Lustre: server umount lustre-MDT0000 complete [ 7079.393393] LustreError: 453790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7079.402891] LustreError: 453790:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7080.165041] LustreError: 455694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784419959 with bad export cookie 5411828683902659670 [ 7080.170911] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7080.171717] LustreError: 455694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7080.312849] Lustre: server umount lustre-MDT0001 complete [ 7082.463363] LNet: 1 local NIs in recovery (showing 1): 192.168.201.145@tcp [ 7099.617814] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 7103.940548] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7117.621299] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:13:16 (1784419996) [ 7126.952878] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7128.539119] Key type lgssc unregistered [ 7128.711743] LNet: 460448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7128.716632] LNetError: 460448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7186.085626] LNet: Removed LNI 192.168.201.145@tcp [ 7186.609283] Key type .llcrypt unregistered [ 7186.611396] Key type ._llcrypt unregistered [ 7195.105322] Key type ._llcrypt registered [ 7195.106723] Key type .llcrypt registered [ 7195.164824] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7195.784524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7195.796119] alg: No test for adler32 (adler32-zlib) [ 7196.674614] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 7196.758698] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 7198.343129] Key type lgssc registered [ 7198.855948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7201.150980] LNet: Added LNI 192.168.201.145@tcp100 [8/256/0/180] [ 7201.602445] LNet: Added LNI 192.168.201.145@tcp101 [8/256/0/180] [ 7202.052607] LNet: Added LNI 192.168.201.145@tcp102 [8/256/0/180] [ 7202.496669] LNet: Added LNI 192.168.201.145@tcp103 [8/256/0/180] [ 7202.950140] LNet: Added LNI 192.168.201.145@tcp104 [8/256/0/180] [ 7203.399458] LNet: Added LNI 192.168.201.145@tcp105 [8/256/0/180] [ 7203.801048] LNet: Added LNI 192.168.201.145@tcp106 [8/256/0/180] [ 7204.234206] LNet: Added LNI 192.168.201.145@tcp107 [8/256/0/180] [ 7204.668691] LNet: Added LNI 192.168.201.145@tcp108 [8/256/0/180] [ 7205.115948] LNet: Added LNI 192.168.201.145@tcp109 [8/256/0/180] [ 7205.547669] LNet: Added LNI 192.168.201.145@tcp110 [8/256/0/180] [ 7205.999287] LNet: Added LNI 192.168.201.145@tcp111 [8/256/0/180] [ 7206.440300] LNet: Added LNI 192.168.201.145@tcp112 [8/256/0/180] [ 7206.871976] LNet: Added LNI 192.168.201.145@tcp113 [8/256/0/180] [ 7207.315360] LNet: Added LNI 192.168.201.145@tcp114 [8/256/0/180] [ 7207.755189] LNet: Added LNI 192.168.201.145@tcp115 [8/256/0/180] [ 7208.182953] LNet: Added LNI 192.168.201.145@tcp116 [8/256/0/180] [ 7208.613306] LNet: Added LNI 192.168.201.145@tcp117 [8/256/0/180] [ 7209.052527] LNet: Added LNI 192.168.201.145@tcp118 [8/256/0/180] [ 7209.486286] LNet: Added LNI 192.168.201.145@tcp119 [8/256/0/180] [ 7209.934683] LNet: Added LNI 192.168.201.145@tcp120 [8/256/0/180] [ 7210.372916] LNet: Added LNI 192.168.201.145@tcp121 [8/256/0/180] [ 7210.818072] LNet: Added LNI 192.168.201.145@tcp122 [8/256/0/180] [ 7211.260448] LNet: Added LNI 192.168.201.145@tcp123 [8/256/0/180] [ 7211.674171] LNet: Added LNI 192.168.201.145@tcp124 [8/256/0/180] [ 7212.100980] LNet: Added LNI 192.168.201.145@tcp125 [8/256/0/180] [ 7212.523368] LNet: Added LNI 192.168.201.145@tcp126 [8/256/0/180] [ 7212.936701] LNet: Added LNI 192.168.201.145@tcp127 [8/256/0/180] [ 7213.376872] LNet: Added LNI 192.168.201.145@tcp128 [8/256/0/180] [ 7213.801733] LNet: Added LNI 192.168.201.145@tcp129 [8/256/0/180] [ 7214.247975] LNet: Added LNI 192.168.201.145@tcp130 [8/256/0/180] [ 7214.690191] LNet: Added LNI 192.168.201.145@tcp131 [8/256/0/180] [ 7215.129251] LNet: Added LNI 192.168.201.145@tcp132 [8/256/0/180] [ 7215.574226] LNet: Added LNI 192.168.201.145@tcp133 [8/256/0/180] [ 7216.011985] LNet: Added LNI 192.168.201.145@tcp134 [8/256/0/180] [ 7216.453234] LNet: Added LNI 192.168.201.145@tcp135 [8/256/0/180] [ 7219.630684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7219.639123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7220.695551] Lustre: 463443:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.145@tcp: can't add NID 192.168.201.145@tcp130: rc = -75 [ 7220.763480] LNetError: 460907:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [ 7220.766159] LNetError: Protocol error connecting to 192.168.201.145@tcp116 at host 192.168.201.145:988: is it running a compatible version of Lustre? [ 7220.830580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7220.843326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7220.884057] Lustre: lustre-MDT0000: new disk, initializing [ 7220.910775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7220.917098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7221.343425] LNet: 5 local NIs in recovery (showing 5): 192.168.201.145@tcp116, 192.168.201.145@tcp122, 192.168.201.145@tcp123, 192.168.201.145@tcp124, 192.168.201.145@tcp121 [ 7221.352918] LNetError: 463470:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [ 7221.358890] LNetError: 463470:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [ 7221.367809] LNetError: Protocol error connecting to 192.168.201.145@tcp116 at host 192.168.201.145:988: is it running a compatible version of Lustre? [ 7221.374276] LNetError: Skipped 6 previous similar messages [ 7222.745888] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7223.401154] LNetError: 460906:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [ 7223.407363] LNetError: 460906:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 7223.412975] LNetError: Protocol error connecting to 192.168.201.145@tcp116 at host 192.168.201.145:988: is it running a compatible version of Lustre? [ 7223.419368] LNetError: Skipped 2 previous similar messages [ 7227.488623] LNetError: 463472:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [ 7227.494724] LNetError: Protocol error connecting to 192.168.201.145@tcp116 at host 192.168.201.145:988: is it running a compatible version of Lustre? [ 7228.231781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.318677] Lustre: 463459:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7228.383928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7228.390343] Lustre: Skipped 1 previous similar message [ 7228.437765] Lustre: lustre-MDT0001: new disk, initializing [ 7228.455502] Lustre: 464262:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.145@tcp: can't add NID 192.168.201.145@tcp130: rc = -75 [ 7228.461250] Lustre: 464262:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7228.533675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7228.558833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7228.565129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7230.463988] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7234.937840] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7235.588495] Lustre: 464908:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.145@tcp: can't add NID 192.168.201.145@tcp130: rc = -75 [ 7235.594064] Lustre: 464908:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7235.680659] LNetError: 463475:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [ 7235.686141] LNetError: Protocol error connecting to 192.168.201.145@tcp116 at host 192.168.201.145:988: is it running a compatible version of Lustre? [ 7237.448376] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7240.683948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7240.847084] Lustre: lustre-OST0000: new disk, initializing [ 7240.850553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7240.855794] Lustre: 465436:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7240.908117] Lustre: 465441:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.145@tcp: can't add NID 192.168.201.145@tcp130: rc = -75 [ 7240.913832] Lustre: 465441:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 7240.943964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7243.454796] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7247.872575] Lustre: 466120:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.145@tcp: can't add NID 192.168.201.145@tcp130: rc = -75 [ 7247.878162] Lustre: 466120:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7247.906470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7247.910698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7247.926306] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7247.963725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7251.040322] LNetError: 463470:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [ 7251.045225] LNetError: Protocol error connecting to 192.168.201.145@tcp116 at host 192.168.201.145:988: is it running a compatible version of Lustre? [ 7252.962772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7252.963588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.967126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7252.972578] Lustre: Skipped 1 previous similar message [ 7256.863007] Lustre: server umount lustre-OST0000 complete [ 7259.104379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7259.109687] LustreError: Skipped 1 previous similar message [ 7259.112799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7259.121878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7259.125473] Lustre: Skipped 1 previous similar message [ 7263.201203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7263.202172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7264.821245] Lustre: server umount lustre-MDT0000 complete [ 7266.340270] LustreError: 464261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784420145 with bad export cookie 18161028880420860457 [ 7266.344122] LustreError: 464261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7266.344291] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7266.468252] Lustre: server umount lustre-MDT0001 complete [ 7270.495805] LNetError: 463464:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.145 [ 7270.498317] LNetError: 463464:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 7270.500233] LNetError: Protocol error connecting to 192.168.201.145@tcp124 at host 192.168.201.145:988: is it running a compatible version of Lustre? [ 7270.502778] LNetError: Skipped 3 previous similar messages [ 7270.954633] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7271.797630] Key type lgssc unregistered [ 7271.926841] LNet: 467347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7271.929455] LNetError: 467347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7271.931673] LNet: Removed LNI 192.168.201.145@tcp [ 7272.991102] LNet: Removed LNI 192.168.201.145@tcp100 [ 7272.992514] LNetError: 467347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7272.994762] LNetError: 467347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7272.997225] LNet: Removed LNI 192.168.201.145@tcp101 [ 7274.015191] LNet: Removed LNI 192.168.201.145@tcp102 [ 7274.017058] LNetError: 467347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7274.019966] LNetError: 467347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7274.023071] LNet: Removed LNI 192.168.201.145@tcp103 [ 7275.039088] LNet: Removed LNI 192.168.201.145@tcp104 [ 7276.063089] LNet: Removed LNI 192.168.201.145@tcp105 [ 7276.064510] LNetError: 467347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7276.066689] LNetError: 467347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7276.069234] LNet: Removed LNI 192.168.201.145@tcp106 [ 7277.087111] LNet: Removed LNI 192.168.201.145@tcp107 [ 7278.111086] LNet: Removed LNI 192.168.201.145@tcp108 [ 7279.135095] LNet: Removed LNI 192.168.201.145@tcp109 [ 7280.159107] LNet: Removed LNI 192.168.201.145@tcp110 [ 7280.161255] LNetError: 467347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7280.165224] LNetError: 467347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7280.169635] LNet: Removed LNI 192.168.201.145@tcp111 [ 7281.183149] LNet: Removed LNI 192.168.201.145@tcp112 [ 7282.207107] LNet: Removed LNI 192.168.201.145@tcp113 [ 7283.231111] LNet: Removed LNI 192.168.201.145@tcp114 [ 7284.255088] LNet: Removed LNI 192.168.201.145@tcp115 [ 7284.256827] LNet: Removed LNI 192.168.201.145@tcp116 [ 7285.279088] LNet: Removed LNI 192.168.201.145@tcp117 [ 7286.303113] LNet: Removed LNI 192.168.201.145@tcp118 [ 7287.327086] LNet: Removed LNI 192.168.201.145@tcp119 [ 7288.351090] LNet: Removed LNI 192.168.201.145@tcp120 [ 7288.352389] LNetError: 467347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7288.354575] LNetError: 467347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 7288.357053] LNet: Removed LNI 192.168.201.145@tcp121 [ 7289.375083] LNet: Removed LNI 192.168.201.145@tcp122 [ 7290.399083] LNet: Removed LNI 192.168.201.145@tcp123 [ 7291.423098] LNet: Removed LNI 192.168.201.145@tcp124 [ 7291.424387] LNet: Removed LNI 192.168.201.145@tcp125 [ 7291.425510] LNet: Removed LNI 192.168.201.145@tcp126 [ 7291.426772] LNet: Removed LNI 192.168.201.145@tcp127 [ 7291.428037] LNet: Removed LNI 192.168.201.145@tcp128 [ 7291.429287] LNet: Removed LNI 192.168.201.145@tcp129 [ 7291.430549] LNet: Removed LNI 192.168.201.145@tcp130 [ 7291.431668] LNet: Removed LNI 192.168.201.145@tcp131 [ 7291.432743] LNet: Removed LNI 192.168.201.145@tcp132 [ 7291.434027] LNet: Removed LNI 192.168.201.145@tcp133 [ 7291.435266] LNet: Removed LNI 192.168.201.145@tcp134 [ 7291.442494] LNet: Removed LNI 192.168.201.145@tcp135 [ 7291.698106] Key type .llcrypt unregistered [ 7291.699590] Key type ._llcrypt unregistered [ 7298.136849] Key type ._llcrypt registered [ 7298.137826] Key type .llcrypt registered [ 7298.171905] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 7303.891074] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7304.231364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7304.241090] alg: No test for adler32 (adler32-zlib) [ 7305.086345] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 7305.168865] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 7306.743082] Key type lgssc registered [ 7307.057589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7317.525129] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:16:36 (1784420196) [ 7322.699865] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7323.644394] Key type lgssc unregistered [ 7323.758409] LNet: 471404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7323.760380] LNetError: 471404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7323.773375] LNet: Removed LNI 192.168.201.145@tcp [ 7324.056262] Key type .llcrypt unregistered [ 7324.057589] Key type ._llcrypt unregistered [ 7330.256892] Key type ._llcrypt registered [ 7330.257868] Key type .llcrypt registered [ 7330.293302] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7330.677317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7330.721989] alg: No test for adler32 (adler32-zlib) [ 7331.573093] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 7331.659225] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 7333.239115] Key type lgssc registered [ 7333.573228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7336.362034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7336.367313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7337.447318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7337.456260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7337.489550] Lustre: lustre-MDT0000: new disk, initializing [ 7337.521935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7337.527211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7338.730813] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7342.501237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7342.542306] Lustre: 472595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7342.569061] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7342.570653] Lustre: Skipped 1 previous similar message [ 7342.603701] Lustre: lustre-MDT0001: new disk, initializing [ 7342.636942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7342.648217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7342.651617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7343.863996] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7347.247874] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7348.819699] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7350.728090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7350.817142] Lustre: lustre-OST0000: new disk, initializing [ 7350.818526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7350.820511] Lustre: 474560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7350.844451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7352.440316] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7355.722961] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7355.893681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7355.896349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7355.924648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7356.689545] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing lnet_if_list [ 7358.063690] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7358.072938] LNet: Added LNI 192.168.201.145@tcp10 [8/256/0/180] [ 7359.621430] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7359.630850] LNet: Added LNI 192.168.201.145@tcp11 [8/256/0/180] [ 7360.784709] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7360.793642] LNet: Added LNI 192.168.201.145@tcp12 [8/256/0/180] [ 7362.026031] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7362.035465] LNet: Added LNI 192.168.201.145@tcp13 [8/256/0/180] [ 7363.284293] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7363.292415] LNet: Added LNI 192.168.201.145@tcp14 [8/256/0/180] [ 7364.549370] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7364.558845] LNet: Added LNI 192.168.201.145@tcp15 [8/256/0/180] [ 7366.092111] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7366.104312] LNet: Added LNI 192.168.201.145@tcp16 [8/256/0/180] [ 7367.489294] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7367.499535] LNet: Added LNI 192.168.201.145@tcp17 [8/256/0/180] [ 7368.885645] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7368.894977] LNet: Added LNI 192.168.201.145@tcp18 [8/256/0/180] [ 7370.320210] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7370.330565] LNet: Added LNI 192.168.201.145@tcp19 [8/256/0/180] [ 7371.758393] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7371.767511] LNet: Added LNI 192.168.201.145@tcp20 [8/256/0/180] [ 7373.397326] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7373.416642] LNet: Added LNI 192.168.201.145@tcp21 [8/256/0/180] [ 7375.852681] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7375.868407] LNet: Added LNI 192.168.201.145@tcp22 [8/256/0/180] [ 7378.347063] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7378.368151] LNet: Added LNI 192.168.201.145@tcp23 [8/256/0/180] [ 7380.903439] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7380.918623] LNet: Added LNI 192.168.201.145@tcp24 [8/256/0/180] [ 7383.497464] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7383.514798] LNet: Added LNI 192.168.201.145@tcp25 [8/256/0/180] [ 7386.330506] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7386.349993] LNet: Added LNI 192.168.201.145@tcp26 [8/256/0/180] [ 7388.936439] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7388.953169] LNet: Added LNI 192.168.201.145@tcp27 [8/256/0/180] [ 7391.495904] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7391.509804] LNet: Added LNI 192.168.201.145@tcp28 [8/256/0/180] [ 7394.387374] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7394.406717] LNet: Added LNI 192.168.201.145@tcp29 [8/256/0/180] [ 7397.223371] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7397.242969] LNet: Added LNI 192.168.201.145@tcp30 [8/256/0/180] [ 7400.016383] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7400.028996] LNet: Added LNI 192.168.201.145@tcp31 [8/256/0/180] [ 7402.983430] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7402.994388] LNet: Added LNI 192.168.201.145@tcp32 [8/256/0/180] [ 7405.920923] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7405.932359] LNet: Added LNI 192.168.201.145@tcp33 [8/256/0/180] [ 7408.908420] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7408.928322] LNet: Added LNI 192.168.201.145@tcp34 [8/256/0/180] [ 7412.343846] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7412.363380] LNet: Added LNI 192.168.201.145@tcp35 [8/256/0/180] [ 7415.343379] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7415.357852] LNet: Added LNI 192.168.201.145@tcp36 [8/256/0/180] [ 7417.944851] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7417.962651] LNet: Added LNI 192.168.201.145@tcp37 [8/256/0/180] [ 7420.714589] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7420.732936] LNet: Added LNI 192.168.201.145@tcp38 [8/256/0/180] [ 7423.757223] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7423.773258] LNet: Added LNI 192.168.201.145@tcp39 [8/256/0/180] [ 7427.050824] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7427.069960] LNet: Added LNI 192.168.201.145@tcp40 [8/256/0/180] [ 7430.052339] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7430.065141] LNet: Added LNI 192.168.201.145@tcp41 [8/256/0/180] [ 7432.910358] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7432.923760] LNet: Added LNI 192.168.201.145@tcp42 [8/256/0/180] [ 7436.340015] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7436.353599] LNet: Added LNI 192.168.201.145@tcp43 [8/256/0/180] [ 7439.748505] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7439.767153] LNet: Added LNI 192.168.201.145@tcp44 [8/256/0/180] [ 7443.119567] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7443.137558] LNet: Added LNI 192.168.201.145@tcp45 [8/256/0/180] [ 7446.092780] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7446.112309] LNet: Added LNI 192.168.201.145@tcp46 [8/256/0/180] [ 7449.325139] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7449.338454] LNet: Added LNI 192.168.201.145@tcp47 [8/256/0/180] [ 7452.879477] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7452.899707] LNet: Added LNI 192.168.201.145@tcp48 [8/256/0/180] [ 7456.671827] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7456.690784] LNet: Added LNI 192.168.201.145@tcp49 [8/256/0/180] [ 7460.801598] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7460.819674] LNet: Added LNI 192.168.201.145@tcp50 [8/256/0/180] [ 7470.560380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7470.565718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7470.574569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7473.634878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7473.638710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7473.645641] Lustre: Skipped 1 previous similar message [ 7475.911268] Lustre: server umount lustre-MDT0000 complete [ 7477.731235] LustreError: 472586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784420356 with bad export cookie 1954907578469990422 [ 7477.739749] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7477.904326] Lustre: server umount lustre-MDT0001 complete [ 7487.780173] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7491.895349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.135126] Lustre: 471869:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784420358/real 1784420358] req@00000000ae9d89d3 x1871100206210816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784420374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7495.148683] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.167239] Lustre: 471866:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784420362/real 1784420362] req@000000001cb107e0 x1871100206210944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784420378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7500.191145] Lustre: 471866:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784420363/real 1784420363] req@00000000420a63b6 x1871100206211072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784420379 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7503.327651] LustreError: 471865:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009476006a x1871100206211968/t0(0) o250->MGC192.168.201.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7503.515345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7504.543170] Lustre: 471868:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784420367/real 1784420367] req@00000000e8273ea6 x1871100206211328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784420383 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7505.539427] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7509.279185] Lustre: 471869:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784420372/real 1784420372] req@000000009f0830b8 x1871100206211456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784420388 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7509.712890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7509.928751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7511.894223] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7514.030230] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7515.382527] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7518.038472] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7529.444367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7529.446977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7529.450057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7529.455799] Lustre: Skipped 1 previous similar message [ 7529.462957] Lustre: Skipped 3 previous similar messages [ 7533.485113] Lustre: server umount lustre-OST0000 complete [ 7535.584319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7535.588810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7535.596579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7539.681483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7539.685985] Lustre: Skipped 1 previous similar message [ 7541.424389] Lustre: server umount lustre-MDT0000 complete [ 7543.205517] LustreError: 474559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784420422 with bad export cookie 1954907578469992508 [ 7543.212267] LustreError: 474559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7543.214226] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7543.384109] Lustre: server umount lustre-MDT0001 complete [ 7548.268388] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7549.814956] Key type lgssc unregistered [ 7550.038747] LNet: 501528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7550.044166] LNetError: 501528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7550.048473] LNet: Removed LNI 192.168.201.145@tcp [ 7550.050806] LNet: Removed LNI 192.168.201.145@tcp10 [ 7550.053209] LNet: Removed LNI 192.168.201.145@tcp11 [ 7550.055560] LNet: Removed LNI 192.168.201.145@tcp12 [ 7550.057938] LNet: Removed LNI 192.168.201.145@tcp13 [ 7550.060341] LNet: Removed LNI 192.168.201.145@tcp14 [ 7550.062765] LNet: Removed LNI 192.168.201.145@tcp15 [ 7550.065147] LNet: Removed LNI 192.168.201.145@tcp16 [ 7550.067495] LNet: Removed LNI 192.168.201.145@tcp17 [ 7550.069919] LNet: Removed LNI 192.168.201.145@tcp18 [ 7550.073153] LNet: Removed LNI 192.168.201.145@tcp19 [ 7550.075467] LNet: Removed LNI 192.168.201.145@tcp20 [ 7550.077849] LNet: Removed LNI 192.168.201.145@tcp21 [ 7550.080210] LNet: Removed LNI 192.168.201.145@tcp22 [ 7550.082546] LNet: Removed LNI 192.168.201.145@tcp23 [ 7550.084763] LNet: Removed LNI 192.168.201.145@tcp24 [ 7550.086952] LNet: Removed LNI 192.168.201.145@tcp25 [ 7550.089133] LNet: Removed LNI 192.168.201.145@tcp26 [ 7550.091331] LNet: Removed LNI 192.168.201.145@tcp27 [ 7550.093491] LNet: Removed LNI 192.168.201.145@tcp28 [ 7550.095577] LNet: Removed LNI 192.168.201.145@tcp29 [ 7550.097481] LNet: Removed LNI 192.168.201.145@tcp30 [ 7550.099445] LNet: Removed LNI 192.168.201.145@tcp31 [ 7550.101482] LNet: Removed LNI 192.168.201.145@tcp32 [ 7550.103479] LNet: Removed LNI 192.168.201.145@tcp33 [ 7550.105393] LNet: Removed LNI 192.168.201.145@tcp34 [ 7550.107232] LNet: Removed LNI 192.168.201.145@tcp35 [ 7550.109123] LNet: Removed LNI 192.168.201.145@tcp36 [ 7550.110948] LNet: Removed LNI 192.168.201.145@tcp37 [ 7550.112806] LNet: Removed LNI 192.168.201.145@tcp38 [ 7550.114534] LNet: Removed LNI 192.168.201.145@tcp39 [ 7550.116198] LNet: Removed LNI 192.168.201.145@tcp40 [ 7550.117900] LNet: Removed LNI 192.168.201.145@tcp41 [ 7550.119522] LNet: Removed LNI 192.168.201.145@tcp42 [ 7550.121248] LNet: Removed LNI 192.168.201.145@tcp43 [ 7550.122862] LNet: Removed LNI 192.168.201.145@tcp44 [ 7550.124376] LNet: Removed LNI 192.168.201.145@tcp45 [ 7550.125969] LNet: Removed LNI 192.168.201.145@tcp46 [ 7550.127469] LNet: Removed LNI 192.168.201.145@tcp47 [ 7550.129065] LNet: Removed LNI 192.168.201.145@tcp48 [ 7550.130389] LNet: Removed LNI 192.168.201.145@tcp49 [ 7550.137562] LNet: Removed LNI 192.168.201.145@tcp50 [ 7550.561243] Key type .llcrypt unregistered [ 7550.563540] Key type ._llcrypt unregistered [ 7559.887333] Key type ._llcrypt registered [ 7559.888748] Key type .llcrypt registered [ 7559.955916] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 7567.719979] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7568.111748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7568.144056] alg: No test for adler32 (adler32-zlib) [ 7569.061723] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 7569.182543] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 7570.767143] Key type lgssc registered [ 7571.392984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7588.001344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7588.010761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.144887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7589.163891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7589.223210] Lustre: lustre-MDT0000: new disk, initializing [ 7589.278436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7589.289387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7591.278336] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7596.749610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7596.805080] Lustre: 504697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7596.842365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7596.847593] Lustre: Skipped 1 previous similar message [ 7596.894598] Lustre: lustre-MDT0001: new disk, initializing [ 7596.932401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7596.947875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7596.951193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7598.593823] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7603.317654] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7605.781515] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7608.811032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7608.876784] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7608.902783] Lustre: lustre-OST0000: new disk, initializing [ 7608.904285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7608.906509] Lustre: 506713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7608.932338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7610.531628] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7613.882500] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7615.477089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7615.480850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7615.512393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7620.577829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7620.578626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7620.580086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7620.581080] Lustre: Skipped 1 previous similar message [ 7620.582871] LustreError: Skipped 1 previous similar message [ 7622.028391] Lustre: server umount lustre-OST0000 complete [ 7625.696685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7625.697369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7625.699644] Lustre: Skipped 1 previous similar message [ 7625.703331] Lustre: Skipped 2 previous similar messages [ 7629.269790] Lustre: server umount lustre-MDT0000 complete [ 7630.500106] LustreError: 505493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784420509 with bad export cookie 9419526332398698099 [ 7630.503695] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7630.504368] LustreError: 505493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7630.610913] Lustre: server umount lustre-MDT0001 complete [ 7633.723981] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7634.557615] Key type lgssc unregistered [ 7634.674400] LNet: 508617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7634.676197] LNetError: 508617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7634.686287] LNet: Removed LNI 192.168.201.145@tcp [ 7634.941512] Key type .llcrypt unregistered [ 7634.942394] Key type ._llcrypt unregistered [ 7641.985982] Key type ._llcrypt registered [ 7641.987039] Key type .llcrypt registered [ 7642.029953] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7642.309311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7642.318507] alg: No test for adler32 (adler32-zlib) [ 7643.203716] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 7643.329250] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 7644.919185] Key type lgssc registered [ 7645.537932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7649.073385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7649.084324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7650.251766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7652.039725] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7656.032837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7656.227749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7658.080777] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7661.405982] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7663.366475] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7665.871321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7666.023159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7668.154390] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7670.853922] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7680.807475] loop0: detected capacity change from 0 to 204800000 [ 7680.956515] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7680.962757] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7680.969221] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7683.932776] loop0: detected capacity change from 0 to 204800000 [ 7684.034085] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7684.039211] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7684.043900] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7686.461870] loop0: detected capacity change from 0 to 204800000 [ 7689.134928] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7689.230820] Lustre: fs15246-OST0000: new disk, initializing [ 7689.234396] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7689.239526] Lustre: 513459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7689.274877] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7692.109815] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7697.365323] loop1: detected capacity change from 0 to 204800000 [ 7700.023981] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7700.077276] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7700.107799] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7700.156570] Lustre: fs15246-MDT0000: new disk, initializing [ 7700.229618] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7700.244915] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7700.306925] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7701.344627] Lustre: 514580:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000002b43a4e8 x1871100533310976/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784420596 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7702.128669] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7710.435551] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7710.441051] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7710.455686] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7715.307816] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7715.315793] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7718.331711] Lustre: server umount fs15246-MDT0000 complete [ 7720.429637] LustreError: 509858:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7720.437401] LustreError: 509858:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7722.652526] Lustre: server umount fs15246-OST0000 complete [ 7728.608389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7728.613034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7728.621120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7730.662909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7730.663729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7733.548298] Lustre: server umount lustre-OST0000 complete [ 7735.778121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.779198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7735.785182] Lustre: Skipped 1 previous similar message [ 7735.787770] Lustre: Skipped 1 previous similar message [ 7740.897921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7740.901829] Lustre: Skipped 2 previous similar messages [ 7741.500955] Lustre: server umount lustre-MDT0000 complete [ 7743.304330] LustreError: 509845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784420622 with bad export cookie 17333241200307882245 [ 7743.310086] LustreError: 509845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7743.312228] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7743.487360] Lustre: server umount lustre-MDT0001 complete [ 7748.342384] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7749.299357] Key type lgssc unregistered [ 7749.418269] LNet: 517674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7749.421054] LNetError: 517674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7749.429362] LNet: Removed LNI 192.168.201.145@tcp [ 7749.757931] Key type .llcrypt unregistered [ 7749.759977] Key type ._llcrypt unregistered [ 7756.620792] Key type ._llcrypt registered [ 7756.621682] Key type .llcrypt registered [ 7756.657280] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7757.013000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7757.024981] alg: No test for adler32 (adler32-zlib) [ 7757.875807] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 7757.972212] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 7759.583110] Key type lgssc registered [ 7759.920906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7777.031619] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 7781.400203] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7798.455214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7798.465588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7799.603301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7799.618912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7799.668274] Lustre: lustre-MDT0000: new disk, initializing [ 7799.703538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7799.714811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7801.696534] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7807.365222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7807.433074] Lustre: 522108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7807.471936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7807.475641] Lustre: Skipped 1 previous similar message [ 7807.527809] Lustre: lustre-MDT0001: new disk, initializing [ 7807.583416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7807.605358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7807.614517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7809.500910] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7814.314910] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7816.505415] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7819.526037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7819.686361] Lustre: lustre-OST0000: new disk, initializing [ 7819.688996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7819.693738] Lustre: 524077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7819.743098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7822.464317] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7827.329412] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7828.988906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7828.992146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7829.031696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7829.169360] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7829.253502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7831.171668] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7831.296426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7834.086243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7834.086315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7834.090663] LustreError: Skipped 1 previous similar message [ 7834.098904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7834.105620] Lustre: Skipped 1 previous similar message [ 7838.314504] Lustre: server umount lustre-OST0000 complete [ 7843.296589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7843.301210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7843.308101] Lustre: Skipped 1 previous similar message [ 7843.311342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7844.321547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7844.322529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7846.262111] Lustre: server umount lustre-MDT0000 complete [ 7848.030542] LustreError: 522101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784420727 with bad export cookie 13665848023797236623 [ 7848.034600] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7848.040749] LustreError: 522101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7848.199275] Lustre: server umount lustre-MDT0001 complete [ 7859.223359] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 7863.742522] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7879.609641] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7733 sec ========== 20:25:58 (1784420758) [ 7880.485247] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:25:59 (1784420759) === [ 7882.147678] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:26:01 (1784420761) === [ 7901.319904] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7902.904070] Key type lgssc unregistered [ 7903.085678] LNet: 530711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7903.090757] LNetError: 530711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7903.103799] LNet: Removed LNI 192.168.201.145@tcp [ 7903.579881] Key type .llcrypt unregistered [ 7903.582103] Key type ._llcrypt unregistered