[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 506286141 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003147] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011153] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014075] *** VALIDATE selinux *** [ 0.022321] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027078] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] *** VALIDATE tmpfs *** [ 0.031392] *** VALIDATE proc *** [ 0.032236] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037024] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041267] debug: unmapping init [mem 0xffffffff8b459000-0xffffffff8b460fff] [ 0.044195] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045676] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050043] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053305] rcu: Hierarchical SRCU implementation. [ 0.055522] smp: Bringing up secondary CPUs ... [ 0.056587] x86: Booting SMP configuration: [ 0.057023] .... node #0, CPUs: #1 #2 #3 [ 0.062188] smp: Brought up 1 node, 4 CPUs [ 0.064021] smpboot: Max logical packages: 1 [ 0.065012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.124093] node 0 deferred pages initialised in 57ms [ 0.126097] devtmpfs: initialized [ 0.127267] x86/mm: Memory block size: 128MB [ 0.131631] gcov: version magic: 0x41383552 [ 0.135294] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136102] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137358] pinctrl core: initialized pinctrl subsystem [ 0.138204] [ 0.138585] ************************************************************* [ 0.139009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140008] ** ** [ 0.141008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142009] ** ** [ 0.143008] ** This means that this kernel is built to expose internal ** [ 0.144021] ** IOMMU data structures, which may compromise security on ** [ 0.145011] ** your system. ** [ 0.146009] ** ** [ 0.147011] ** If you see this message and you are not debugging the ** [ 0.148011] ** kernel, report this immediately to your vendor! ** [ 0.149010] ** ** [ 0.150010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151015] ************************************************************* [ 0.152757] NET: Registered protocol family 16 [ 0.153490] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.154079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.157074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.162035] cpuidle: using governor menu [ 0.165375] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.169491] PCI: Using configuration type 1 for base access [ 0.172169] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189069] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193101] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197050] cryptd: max_cpu_qlen set to 1000 [ 0.200277] ACPI: Added _OSI(Module Device) [ 0.202013] ACPI: Added _OSI(Processor Device) [ 0.204013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213844] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.223346] ACPI: Interpreter enabled [ 0.225058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.226013] ACPI: Using IOAPIC for interrupt routing [ 0.228131] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.232322] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242585] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.246046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.249021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.252084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258522] acpiphp: Slot [2] registered [ 0.260123] acpiphp: Slot [5] registered [ 0.262123] acpiphp: Slot [6] registered [ 0.263134] acpiphp: Slot [7] registered [ 0.265188] acpiphp: Slot [8] registered [ 0.267127] acpiphp: Slot [9] registered [ 0.268119] acpiphp: Slot [10] registered [ 0.270119] acpiphp: Slot [3] registered [ 0.272100] acpiphp: Slot [4] registered [ 0.273108] acpiphp: Slot [11] registered [ 0.275128] acpiphp: Slot [12] registered [ 0.277103] acpiphp: Slot [13] registered [ 0.278103] acpiphp: Slot [14] registered [ 0.279100] acpiphp: Slot [15] registered [ 0.281101] acpiphp: Slot [16] registered [ 0.283108] acpiphp: Slot [17] registered [ 0.284144] acpiphp: Slot [18] registered [ 0.286096] acpiphp: Slot [19] registered [ 0.288106] acpiphp: Slot [20] registered [ 0.289083] acpiphp: Slot [21] registered [ 0.290070] acpiphp: Slot [22] registered [ 0.292107] acpiphp: Slot [23] registered [ 0.293156] acpiphp: Slot [24] registered [ 0.294074] acpiphp: Slot [25] registered [ 0.296070] acpiphp: Slot [26] registered [ 0.297075] acpiphp: Slot [27] registered [ 0.298099] acpiphp: Slot [28] registered [ 0.300110] acpiphp: Slot [29] registered [ 0.302124] acpiphp: Slot [30] registered [ 0.303098] acpiphp: Slot [31] registered [ 0.305067] PCI host bridge to bus 0000:00 [ 0.306018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.309025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.312023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.315012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.317027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.321028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.322180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325095] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.329272] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.344867] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.345000] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.346031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.349020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.352210] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.354613] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.357032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.360366] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.366015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.379017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.384014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.389141] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.397017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.405015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.424017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.431461] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.439017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.453017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.476015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.489169] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.500019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.506015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.535019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.548448] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.561018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.576041] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.600024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.611808] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.618018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.626017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.644016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.654084] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.661016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.669022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.688019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.699232] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.701347] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.704363] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.707467] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.709205] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.714130] iommu: Default domain type: Passthrough [ 0.716662] SCSI subsystem initialized [ 0.718142] ACPI: bus type USB registered [ 0.720089] usbcore: registered new interface driver usbfs [ 0.722060] usbcore: registered new interface driver hub [ 0.723090] usbcore: registered new device driver usb [ 0.725128] pps_core: LinuxPPS API ver. 1 registered [ 0.727011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.728052] PTP clock support registered [ 0.730151] EDAC MC: Ver: 3.0.0 [ 0.732155] PCI: Using ACPI for IRQ routing [ 0.733767] NetLabel: Initializing [ 0.735007] NetLabel: domain hash size = 128 [ 0.736008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.738084] NetLabel: unlabeled traffic allowed by default [ 0.741142] vgaarb: loaded [ 0.743264] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.745013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.750292] clocksource: Switched to clocksource kvm-clock [ 0.871229] VFS: Disk quotas dquot_6.6.0 [ 0.873022] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.875372] *** VALIDATE ramfs *** [ 0.876761] *** VALIDATE hugetlbfs *** [ 0.878426] pnp: PnP ACPI init [ 0.880834] pnp: PnP ACPI: found 6 devices [ 0.898393] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.901960] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.904095] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.906300] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.908841] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.911508] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.914575] NET: Registered protocol family 2 [ 0.917123] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.921883] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.925659] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.931276] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.934809] TCP: Hash tables configured (established 65536 bind 65536) [ 0.937737] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.940736] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.943422] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.946043] NET: Registered protocol family 1 [ 0.948242] RPC: Registered named UNIX socket transport module. [ 0.950253] RPC: Registered udp transport module. [ 0.951856] RPC: Registered tcp transport module. [ 0.953432] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.955626] NET: Registered protocol family 44 [ 0.956990] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.958997] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.961056] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.963257] PCI: CLS 0 bytes, default 64 [ 0.964889] Unpacking initramfs... [ 2.342710] debug: unmapping init [mem 0xffff997ebcc54000-0xffff997ebffbffff] [ 2.346540] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.348970] software IO TLB: mapped [mem 0x00000000a1400000-0x00000000a5400000] (64MB) [ 2.351969] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.852944] Initialise system trusted keyrings [ 2.854863] Key type blacklist registered [ 2.856873] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.864322] zbud: loaded [ 2.867266] *** VALIDATE nfs *** [ 2.868578] *** VALIDATE nfs4 *** [ 2.870169] pstore: using deflate compression [ 2.874494] Platform Keyring initialized [ 2.980708] NET: Registered protocol family 38 [ 2.982851] Key type asymmetric registered [ 2.984564] Asymmetric key parser 'x509' registered [ 2.986768] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.990402] io scheduler mq-deadline registered [ 2.994602] io scheduler kyber registered [ 2.996341] io scheduler bfq registered [ 3.000565] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.003612] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.006245] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.009662] ACPI: Power Button [PWRF] [ 3.016827] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.025419] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.040292] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.048115] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.060586] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.090294] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.120437] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.127030] Non-volatile memory driver v1.3 [ 3.128775] Linux agpgart interface v0.103 [ 3.161445] virtio_blk virtio1: [vda] 147096 512-byte logical blocks (75.3 MB/71.8 MiB) [ 3.164334] vda: detected capacity change from 0 to 75313152 [ 3.177264] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.180229] vdb: detected capacity change from 0 to 1073741824 [ 3.194408] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.197929] vdc: detected capacity change from 0 to 2621440000 [ 3.211194] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.214179] vdd: detected capacity change from 0 to 2621440000 [ 3.229786] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.233197] vde: detected capacity change from 0 to 4294967296 [ 3.252462] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.254917] vdf: detected capacity change from 0 to 4294967296 [ 3.264197] libphy: Fixed MDIO Bus: probed [ 3.273585] usbcore: registered new interface driver usbserial_generic [ 3.275222] usbserial: USB Serial support registered for generic [ 3.276927] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.279682] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.280939] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.282402] mousedev: PS/2 mouse device common for all mice [ 3.284794] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.288587] rtc_cmos 00:05: RTC can wake from S4 [ 3.290885] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.295454] rtc_cmos 00:05: registered as rtc0 [ 3.296568] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.297464] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.303490] intel_pstate: CPU model not supported [ 3.306975] hid: raw HID events driver (C) Jiri Kosina [ 3.309199] usbcore: registered new interface driver usbhid [ 3.311222] usbhid: USB HID core driver [ 3.312751] drop_monitor: Initializing network drop monitor service [ 3.315166] Initializing XFRM netlink socket [ 3.316972] NET: Registered protocol family 10 [ 3.319906] Segment Routing with IPv6 [ 3.321322] NET: Registered protocol family 17 [ 3.323587] mpls_gso: MPLS GSO support [ 3.329231] RAS: Correctable Errors collector initialized. [ 3.331452] AVX version of gcm_enc/dec engaged. [ 3.333218] AES CTR mode by8 optimization enabled [ 3.403939] sched_clock: Marking stable (3403916451, 0)->(4320365223, -916448772) [ 3.407461] registered taskstats version 1 [ 3.409156] Loading compiled-in X.509 certificates [ 3.411045] zswap: loaded using pool lzo/zbud [ 3.435570] Key type big_key registered [ 3.447603] Key type encrypted registered [ 3.449060] ima: No TPM chip found, activating TPM-bypass! [ 3.451069] ima: Allocated hash algorithm: sha1 [ 3.452650] ima: No architecture policies found [ 3.453695] evm: Initialising EVM extended attributes: [ 3.455674] evm: security.selinux [ 3.456748] evm: security.ima [ 3.457250] evm: security.capability [ 3.458218] evm: HMAC attrs: 0x1 [ 3.459980] rtc_cmos 00:05: setting system clock to 2026-08-27 19:41:36 UTC (1787859696) [ 3.465368] debug: unmapping init [mem 0xffffffff8c403000-0xffffffff8c5fffff] [ 3.467493] debug: unmapping init [mem 0xffffffff8b182000-0xffffffff8b458fff] [ 3.474132] Write protecting the kernel read-only data: 28672k [ 3.477405] debug: unmapping init [mem 0xffffffff89803000-0xffffffff899fffff] [ 3.480554] debug: unmapping init [mem 0xffffffff8a114000-0xffffffff8a1fffff] [ 3.511965] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.517326] systemd[1]: Detected virtualization kvm. [ 3.518806] systemd[1]: Detected architecture x86-64. [ 3.520027] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.546306] systemd[1]: No hostname configured. [ 3.548131] systemd[1]: Set hostname to . [ 3.550306] random: systemd: uninitialized urandom read (16 bytes read) [ 3.552895] systemd[1]: Initializing machine ID from random generator. [ 3.685939] random: systemd: uninitialized urandom read (16 bytes read) [ 3.688946] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.693269] random: systemd: uninitialized urandom read (16 bytes read) [ 3.695984] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.699338] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Apply Kernel Variables... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.257569] device-mapper: uevent: version 1.0.3 [ 4.259443] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.944123] virtio_net virtio0 ens2: renamed from eth0 [ 4.951117] random: fast init done [ 5.009601] scsi host0: ata_piix [ 5.065156] scsi host1: ata_piix [ 5.066883] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.069145] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.787304] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.963783] random: crng init done [ 9.965214] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.371828] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.616135] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.861091] SELinux: Disabled at runtime. [ 11.962676] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.972265] systemd[1]: Detected virtualization kvm. [ 11.974300] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.451775] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.455385] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.459889] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.463937] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.468091] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.477978] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.488102] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target I[ 12.641777] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS nitrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.948256] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.258850] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.329195] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.487856] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.501503] EDAC sbridge: Ver: 1.1.2 [ 15.325683] Key type dns_resolver registered [ 15.619633] NFS: Registering the id_resolver key type [ 15.621541] Key type id_resolver registered [ 15.622933] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg231-server login: [ 52.304008] hrtimer: interrupt took 3301401 ns [ 56.178228] libcfs: loading out-of-tree module taints kernel. [ 56.252946] Key type ._llcrypt registered [ 56.255688] Key type .llcrypt registered [ 56.374107] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 77.839532] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 80.519464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 80.560740] alg: No test for adler32 (adler32-zlib) [ 82.568205] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 83.719416] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 85.562376] Key type lgssc registered [ 88.450040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.335814] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.857923] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 185.177496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 185.238276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.704257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 186.749504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 186.916594] Lustre: lustre-MDT0000: new disk, initializing [ 187.049082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.083951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 192.691290] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.138984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.228147] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 209.269638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 209.272444] Lustre: Skipped 1 previous similar message [ 209.466385] Lustre: lustre-MDT0001: new disk, initializing [ 209.589777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.653792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 209.677617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.164785] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.764764] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 232.681324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.922891] Lustre: lustre-OST0000: new disk, initializing [ 232.926672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 232.932870] Lustre: 8456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.997037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.447564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 235.458575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 235.565533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 240.780333] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.280465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 259.460500] Lustre: lustre-OST0001: new disk, initializing [ 259.465746] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 259.472660] Lustre: 9531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 259.544712] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 267.076460] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.420266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 269.430971] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 269.573069] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 281.191691] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 290.257409] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 298.931401] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing check_logdir /tmp/testlogs/ [ 304.733933] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing yml_node [ 310.207615] Lustre: DEBUG MARKER: Client: 2.17.57.124 [ 313.342502] Lustre: DEBUG MARKER: MDS: 2.17.57.124 [ 317.204560] Lustre: DEBUG MARKER: OSS: 2.17.57.124 [ 319.266633] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 15:46:51 EDT 2026 [ 340.605298] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 342.451625] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 361.955473] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.959293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.973980] Lustre: Skipped 1 previous similar message [ 361.987445] Lustre: Skipped 3 previous similar messages [ 366.534425] Lustre: server umount lustre-MDT0000 complete [ 370.901822] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860063 with bad export cookie 14006580117295800492 [ 370.911419] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 370.916448] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 372.192691] LustreError: 6525:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.201122] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.213274] LustreError: 6525:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 372.234076] Lustre: Skipped 2 previous similar messages [ 372.237655] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.243016] Lustre: Skipped 1 previous similar message [ 375.202966] LustreError: 6526:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.205420] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 375.240699] Lustre: Skipped 1 previous similar message [ 377.401992] Lustre: server umount lustre-MDT0001 complete [ 381.991317] Lustre: server umount lustre-OST0000 complete [ 385.921656] Lustre: server umount lustre-OST0001 complete [ 394.061030] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 405.577960] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 441.408773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.594170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 441.619177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 441.689983] Lustre: lustre-MDT0000: new disk, initializing [ 441.767669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.788302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 446.023976] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 458.123983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.319084] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 458.425493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 458.430277] Lustre: Skipped 1 previous similar message [ 458.576257] Lustre: lustre-MDT0001: new disk, initializing [ 458.712072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 458.771882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 458.808794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 464.339659] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 475.897941] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 482.316713] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 490.714664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 491.305776] Lustre: lustre-OST0000: new disk, initializing [ 491.317605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 491.340395] Lustre: 17055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 491.586691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.568775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 492.576991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 492.775794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 500.256760] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 510.419477] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 514.098384] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 514.411971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 520.924738] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 521.264074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 523.748317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.775415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 523.777313] Lustre: Skipped 1 previous similar message [ 523.799212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 523.826298] Lustre: Skipped 1 previous similar message [ 528.881843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.887252] Lustre: Skipped 1 previous similar message [ 530.007725] Lustre: server umount lustre-OST0000 complete [ 535.524111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 535.549153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.578564] Lustre: Skipped 1 previous similar message [ 539.116577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.125884] Lustre: Skipped 2 previous similar messages [ 541.116804] Lustre: server umount lustre-MDT0000 complete [ 544.230175] LustreError: 15094:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.261919] LustreError: 15094:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 545.771983] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860238 with bad export cookie 14006580117295802025 [ 545.773681] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 545.779916] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 546.202131] Lustre: server umount lustre-MDT0001 complete [ 552.294591] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 15:50:43 (1787860243) [ 555.299290] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 557.171546] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 15:50:49 (1787860249) [ 564.685413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.091506] LustreError: 19467:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.182889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 570.340961] LustreError: 19468:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.412461] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 579.629536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.499498] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 594.228681] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 600.782107] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 607.996725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 615.506481] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 623.981483] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 635.365993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 635.371185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.389312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 635.404779] Lustre: Skipped 2 previous similar messages [ 635.425708] Lustre: Skipped 1 previous similar message [ 639.744093] Lustre: server umount lustre-OST0000 complete [ 645.612355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.640689] Lustre: Skipped 1 previous similar message [ 649.885271] Lustre: server umount lustre-MDT0000 complete [ 650.722280] LustreError: 20195:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.734948] LustreError: 20195:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 654.070089] LustreError: 19447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860347 with bad export cookie 14006580117295802907 [ 654.081200] LustreError: 19447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 654.082826] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 654.476917] Lustre: server umount lustre-MDT0001 complete [ 668.545160] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 672.021208] Key type lgssc unregistered [ 672.329934] LNet: 23005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.344556] LNetError: 23005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.367451] LNet: Removed LNI 192.168.202.131@tcp [ 673.673595] Key type .llcrypt unregistered [ 673.676090] Key type ._llcrypt unregistered [ 698.182561] Key type ._llcrypt registered [ 698.185174] Key type .llcrypt registered [ 698.292541] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 699.660706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 699.769954] alg: No test for adler32 (adler32-zlib) [ 700.939728] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 701.203116] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 702.880586] Key type lgssc registered [ 704.223360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 712.915261] loop: module loaded [ 713.022731] loop0: detected capacity change from 0 to 204800000 [ 713.285864] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 713.307450] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 713.743465] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 721.010642] loop0: detected capacity change from 0 to 204800000 [ 721.215468] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 721.258672] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 721.273865] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 727.529507] loop0: detected capacity change from 0 to 204800000 [ 734.755901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 734.810223] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.176228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 736.284548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 736.358825] Lustre: lustre-MDT0000: new disk, initializing [ 736.483635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 736.548156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 741.878975] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 751.870541] loop1: detected capacity change from 0 to 204800000 [ 758.517182] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 758.827167] Lustre: lustre-OST07c6: new disk, initializing [ 758.830804] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 758.835615] Lustre: Skipped 1 previous similar message [ 758.840233] Lustre: 26200:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 758.905290] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 766.268484] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 769.139984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 769.155290] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 769.193475] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 771.584699] Lustre: 25184:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 771.592289] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 771.597933] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 771.603312] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 771.610693] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 771.614695] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 787.707595] Lustre: 25186:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 787.724307] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 787.734812] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 787.751085] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 787.765477] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 787.780501] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 792.897198] Lustre: 25186:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 792.910666] Lustre: 25186:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 792.924655] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 792.935489] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.950629] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 792.960899] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.971679] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 792.984768] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.994023] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 793.011578] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.023429] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.040594] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.808074] Lustre: 25184:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 797.822849] Lustre: 25184:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 797.834276] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 797.845839] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.868479] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 797.874820] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.884826] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 797.893742] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.906357] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 797.921751] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.939038] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 797.951685] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.951451] Lustre: 25186:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 805.958072] Lustre: 25186:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages [ 805.963215] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 805.967430] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 805.974237] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 805.978705] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 805.983669] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 805.988583] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 805.992354] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 805.997084] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.001380] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 806.003747] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 817.208915] Lustre: 25186:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 817.216797] Lustre: 25186:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages [ 817.223741] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 817.233451] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 817.246258] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 817.256206] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 817.265191] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 817.273403] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 817.279320] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 817.283550] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 817.287859] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 817.291644] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 866.283946] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 866.305289] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 869.290344] Lustre: server umount lustre-OST07c6 complete [ 880.527644] Lustre: server umount lustre-MDT0000 complete [ 898.946690] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 902.127920] Key type lgssc unregistered [ 902.393719] LNet: 28761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.402367] LNetError: 28761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.418842] LNet: Removed LNI 192.168.202.131@tcp [ 903.318324] Key type .llcrypt unregistered [ 903.321136] Key type ._llcrypt unregistered [ 924.723936] Key type ._llcrypt registered [ 924.732503] Key type .llcrypt registered [ 924.871449] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 926.460315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.478243] alg: No test for adler32 (adler32-zlib) [ 927.620246] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 927.830539] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 929.560299] Key type lgssc registered [ 931.067619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 939.441991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 939.467257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.898860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.187818] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 950.249467] LustreError: 30034:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 950.267366] LustreError: 30034:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 955.358153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.361737] LustreError: 30035:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 955.715823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 959.870395] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 969.186749] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 976.420791] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 983.761670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 984.107780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.163952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 991.141846] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 999.123440] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1015.781751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.797636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1015.802590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1015.816284] Lustre: Skipped 1 previous similar message [ 1016.894327] Lustre: server umount lustre-OST0000 complete [ 1022.441112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1022.448356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.474254] Lustre: Skipped 1 previous similar message [ 1022.504504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.017352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.029133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.477125] Lustre: server umount lustre-MDT0000 complete [ 1031.137166] LustreError: 30034:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1031.148290] LustreError: 30034:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1031.293353] LustreError: 30017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860724 with bad export cookie 9562126436589947685 [ 1031.294293] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1031.305325] LustreError: 30017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1031.656231] Lustre: server umount lustre-MDT0001 complete [ 1043.792817] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1047.059206] Key type lgssc unregistered [ 1047.397917] LNet: 33777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.402250] LNetError: 33777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.433111] LNet: Removed LNI 192.168.202.131@tcp [ 1048.764421] Key type .llcrypt unregistered [ 1048.766266] Key type ._llcrypt unregistered [ 1067.713960] Key type ._llcrypt registered [ 1067.716845] Key type .llcrypt registered [ 1067.859420] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1069.767269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1069.807220] alg: No test for adler32 (adler32-zlib) [ 1071.176474] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 1071.486490] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1073.224184] Key type lgssc registered [ 1074.886461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1084.870593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1084.893529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.653276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.761323] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1096.173853] LustreError: 35048:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1096.188481] LustreError: 35048:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1101.283741] LustreError: 35049:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.542439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.076629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1108.696795] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1116.554313] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1123.942621] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1131.066589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1131.691655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1139.930789] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1148.159274] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1156.298345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1168.865482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1168.874187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.901733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1171.940727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1171.949623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1171.960653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.095980] Lustre: server umount lustre-OST0000 complete [ 1180.144127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1180.152739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.164410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.167239] Lustre: Skipped 1 previous similar message [ 1182.177165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1182.179095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1185.541703] Lustre: server umount lustre-MDT0000 complete [ 1187.302578] LustreError: 35048:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.314105] LustreError: 35048:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1189.349157] LustreError: 35030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860882 with bad export cookie 2664007663708313813 [ 1189.362164] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1189.363977] LustreError: 35030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1190.122532] Lustre: server umount lustre-MDT0001 complete [ 1202.820252] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1206.140488] Key type lgssc unregistered [ 1206.307851] LNet: 38622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.313788] LNetError: 38622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1206.331913] LNet: Removed LNI 192.168.202.131@tcp [ 1207.172644] Key type .llcrypt unregistered [ 1207.175336] Key type ._llcrypt unregistered [ 1236.764900] Key type ._llcrypt registered [ 1236.767713] Key type .llcrypt registered [ 1236.854740] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1238.003426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1238.308430] alg: No test for adler32 (adler32-zlib) [ 1239.533867] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 1239.927365] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1241.696984] Key type lgssc registered [ 1243.058545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1252.776581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1252.822613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1254.328312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1260.122176] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1264.616746] LustreError: 40045:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.647991] LustreError: 40045:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1269.741818] LustreError: 40044:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.403566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1270.861923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1276.998336] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1286.042906] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1292.648422] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1301.054931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1301.459539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1308.478763] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1315.804776] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1325.072937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1335.269460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.281366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1335.308644] Lustre: Skipped 2 previous similar messages [ 1335.333565] Lustre: Skipped 2 previous similar messages [ 1336.743877] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 1339.774206] Lustre: server umount lustre-MDT0000 complete [ 1340.403696] LustreError: 40040:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.439801] LustreError: 40040:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1344.151246] LustreError: 40024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861037 with bad export cookie 6637638429778671380 [ 1344.159158] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1344.164838] LustreError: 40024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1345.510090] LustreError: 40040:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.510469] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1345.547240] LustreError: 40040:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1345.606346] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1349.538034] LustreError: 40041:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.545064] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1349.559541] LustreError: 40041:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1349.590130] Lustre: Skipped 1 previous similar message [ 1350.828811] Lustre: server umount lustre-MDT0001 complete [ 1367.805382] Lustre: server umount lustre-OST0000 complete [ 1387.574807] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1390.714681] Key type lgssc unregistered [ 1391.030854] LNet: 43933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.053847] LNetError: 43933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1391.068878] LNet: Removed LNI 192.168.202.131@tcp [ 1392.118923] Key type .llcrypt unregistered [ 1392.124068] Key type ._llcrypt unregistered [ 1421.442778] Key type ._llcrypt registered [ 1421.448908] Key type .llcrypt registered [ 1421.687698] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1423.442865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1423.620675] alg: No test for adler32 (adler32-zlib) [ 1425.179374] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 1425.539919] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1427.258393] Key type lgssc registered [ 1428.588892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1437.711990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1437.750263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1439.292127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1444.817782] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1449.452837] LustreError: 45352:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.464935] LustreError: 45352:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1454.573430] LustreError: 45351:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.127612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1455.633378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1460.803573] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1469.544667] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1475.852629] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1483.176080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1483.708571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.745522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1491.361685] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1499.179868] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1505.250777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1505.262421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1505.265228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1505.271305] LustreError: Skipped 1 previous similar message [ 1505.303439] Lustre: Skipped 1 previous similar message [ 1506.224156] Lustre: lustre-OST0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 1506.234960] Lustre: Skipped 1 previous similar message [ 1510.372202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1515.515074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1515.533032] Lustre: Skipped 3 previous similar messages [ 1519.072106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1519.234323] Lustre: server umount lustre-OST0000 complete [ 1541.093849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.114594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1541.118832] Lustre: Skipped 2 previous similar messages [ 1544.746150] Lustre: server umount lustre-MDT0000 complete [ 1545.270768] LustreError: 45348:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1545.302351] LustreError: 45348:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1548.692914] LustreError: 45333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861241 with bad export cookie 5411729205479228856 [ 1548.703641] LustreError: 45333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1548.705641] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1555.108724] Lustre: server umount lustre-MDT0001 complete [ 1567.727069] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1570.791174] Key type lgssc unregistered [ 1571.119893] LNet: 49064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.126731] LNetError: 49064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.145527] LNet: Removed LNI 192.168.202.131@tcp [ 1572.192199] Key type .llcrypt unregistered [ 1572.194767] Key type ._llcrypt unregistered [ 1600.476245] Key type ._llcrypt registered [ 1600.481514] Key type .llcrypt registered [ 1600.618372] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1601.865494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1601.884227] alg: No test for adler32 (adler32-zlib) [ 1603.043306] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 1603.273667] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1605.040316] Key type lgssc registered [ 1606.504928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1615.481141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1615.505059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1617.047970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1622.732379] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1627.623847] LustreError: 50481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1627.656172] LustreError: 50481:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1632.741988] LustreError: 50482:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.186212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.649869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1638.085848] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1646.331148] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1654.008883] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1662.493096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1663.136390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.187806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1670.985260] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1678.921807] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1685.631955] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1689.889096] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1702.901994] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2716h09m53s [ 1704.912676] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1707.422678] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 1708.477507] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 1709.545804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.561204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.578267] Lustre: Skipped 2 previous similar messages [ 1709.611959] Lustre: Skipped 2 previous similar messages [ 1713.514071] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 1713.524689] Lustre: Skipped 3 previous similar messages [ 1713.757759] Lustre: server umount lustre-MDT0000 complete [ 1716.714622] LustreError: 50478:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.747231] LustreError: 50478:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1717.557871] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1719.667525] LustreError: 50477:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.118132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.293406] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1725.746422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.885381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1726.768295] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1731.557786] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1735.541917] Lustre: DEBUG MARKER: Wait for df (36750) ... [ 1737.407720] Lustre: DEBUG MARKER: done [ 1742.311311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.341705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1742.350685] Lustre: Skipped 1 previous similar message [ 1752.546892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1752.562621] Lustre: Skipped 3 previous similar messages [ 1756.129117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1756.349840] Lustre: server umount lustre-OST0000 complete [ 1762.788214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1762.809151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1762.819361] Lustre: Skipped 2 previous similar messages [ 1767.619774] Lustre: server umount lustre-MDT0000 complete [ 1769.960914] LustreError: 50481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1769.982673] LustreError: 50481:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 1771.593473] LustreError: 50462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861464 with bad export cookie 1399253776643311089 [ 1771.595524] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1771.604471] LustreError: 50462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1771.939215] Lustre: server umount lustre-MDT0001 complete [ 1782.594558] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1784.988202] Key type lgssc unregistered [ 1785.259657] LNet: 55423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1785.268640] LNetError: 55423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1785.284900] LNet: Removed LNI 192.168.202.131@tcp [ 1786.098455] Key type .llcrypt unregistered [ 1786.101343] Key type ._llcrypt unregistered [ 1837.531770] Key type ._llcrypt registered [ 1837.535769] Key type .llcrypt registered [ 1837.755447] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1839.090477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1839.136115] alg: No test for adler32 (adler32-zlib) [ 1840.436463] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 1840.626264] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1842.352557] Key type lgssc registered [ 1843.215855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1851.114742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1851.135676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1852.551239] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1852.566257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1852.752716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1857.890462] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1867.407427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.603555] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1867.694827] Lustre: 57632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1868.235728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1873.876728] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1882.561877] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1888.716593] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1897.672926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1897.988460] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1898.173779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.955827] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1905.810572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1913.065661] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1920.558781] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2716h13m32s [ 1921.903870] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1929.914531] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1932.649048] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1932.718729] LustreError: 59783:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1953.162162] Lustre: lustre-MDT0000: Client 5b896f4e-2859-4f35-9643-3a21b363db4b (at 192.168.202.31@tcp) reconnecting [ 1972.747382] LustreError: 59783:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1975.263447] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1978.051977] Lustre: DEBUG MARKER: done [ 1987.553652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1987.561637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.575218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1988.076322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1988.081267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.081435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1990.705576] Lustre: server umount lustre-OST0000 complete [ 1996.259486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1996.266712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.278458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1996.293926] Lustre: Skipped 1 previous similar message [ 1999.400376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1999.412188] Lustre: Skipped 4 previous similar messages [ 2000.687909] Lustre: server umount lustre-MDT0000 complete [ 2005.708561] LustreError: 57626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861698 with bad export cookie 11913304909645073624 [ 2005.715744] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2005.723792] LustreError: 57626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2006.107144] Lustre: server umount lustre-MDT0001 complete [ 2020.107537] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2023.158439] Key type lgssc unregistered [ 2023.568426] LNet: 61969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.576293] LNetError: 61969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.601083] LNet: Removed LNI 192.168.202.131@tcp [ 2024.599580] Key type .llcrypt unregistered [ 2024.603273] Key type ._llcrypt unregistered [ 2071.780640] Key type ._llcrypt registered [ 2071.783307] Key type .llcrypt registered [ 2071.923717] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2073.061963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2073.080660] alg: No test for adler32 (adler32-zlib) [ 2074.159708] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 2074.462547] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2076.144351] Key type lgssc registered [ 2077.495759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2081.021835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2081.061690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.362101] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2082.402452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2082.618315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.478437] Lustre: Failing over lustre-MDT0000 [ 2083.803229] Lustre: server umount lustre-MDT0000 complete [ 2090.912607] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:16:23 (1787861783) [ 2097.780871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2098.099518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.426209] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2111.926920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.003263] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2112.051682] Lustre: 64772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2112.187309] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2112.389699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2118.338438] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2126.548660] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2132.828362] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2143.431484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2143.727026] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2144.097918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.647860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2152.721950] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2161.815537] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2167.664833] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2177.505736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.511735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2177.529667] Lustre: Skipped 1 previous similar message [ 2177.551252] Lustre: Skipped 2 previous similar messages [ 2182.628769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.636780] Lustre: Skipped 2 previous similar messages [ 2187.746105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.767508] Lustre: Skipped 2 previous similar messages [ 2189.280128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2189.498173] Lustre: server umount lustre-MDT0000 complete [ 2193.185290] LustreError: 66626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861886 with bad export cookie 11258725006608304788 [ 2193.192859] LustreError: 66626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2193.193944] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2193.509242] Lustre: server umount lustre-MDT0001 complete [ 2197.415531] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2202.980375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2212.370876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.368160] Lustre: 63691:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861891/real 1787861891] req@0000000069b33397 x1874708940239360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787861907 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2214.402218] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.412429] Lustre: Skipped 1 previous similar message [ 2215.397233] Lustre: 63693:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861892/real 1787861892] req@00000000dfed249b x1874708940239488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787861908 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2218.529125] LustreError: 63690:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004a876ea9 x1874708940240384/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2218.530936] Lustre: 63693:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861895/real 1787861895] req@00000000d7f245e7 x1874708940239616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787861911 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2219.028294] LustreError: 66620:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2219.059161] LustreError: 66620:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2219.172958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2219.270255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2223.269406] LustreError: 66621:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.287536] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2223.564935] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2224.289439] Lustre: 63691:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861901/real 1787861901] req@0000000093c9a98a x1874708940240000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787861917 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2224.313595] Lustre: 63691:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2224.358366] LustreError: 68540:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.379214] LustreError: 68540:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2229.217760] Lustre: 63694:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861906/real 1787861906] req@000000002f5495db x1874708940240128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787861922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2229.738786] LustreError: 68543:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.755737] LustreError: 68543:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2231.740558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2231.918174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2235.726886] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2237.443636] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2244.271954] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2250.193792] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2257.181255] Lustre: server umount lustre-MDT0000 complete [ 2257.381854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2257.390558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.408507] LustreError: 66620:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2262.511213] LustreError: 64764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861955 with bad export cookie 11258725006608307070 [ 2262.523711] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.526970] LustreError: 64764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2263.013151] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.027018] Lustre: Skipped 2 previous similar messages [ 2263.213184] Lustre: server umount lustre-MDT0001 complete [ 2272.951440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2284.931325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.072345] LustreError: 71211:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2287.096660] LustreError: 71211:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000897584df x1874708940265600/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787861980 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2287.144582] LustreError: 71211:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2288.091479] LustreError: 66621:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.127252] LustreError: 66621:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2288.221393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.342349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2294.714578] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2304.018949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.032380] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2310.123731] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2312.186615] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2318.109695] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2325.145471] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2332.478883] Lustre: server umount lustre-MDT0000 complete [ 2332.651231] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.655056] LustreError: 66620:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.667530] Lustre: Skipped 1 previous similar message [ 2332.692420] LustreError: 66620:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 2336.312437] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862029 with bad export cookie 11258725006608308449 [ 2336.316115] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2336.322232] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2336.569813] Lustre: server umount lustre-MDT0001 complete [ 2343.626391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2347.470047] Lustre: DEBUG MARKER: files compared the same [ 2360.064234] Lustre: server umount lustre-OST0000 complete [ 2380.609217] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2383.700356] Key type lgssc unregistered [ 2383.959455] LNet: 74504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2383.973726] LNetError: 74504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2383.984752] LNet: Removed LNI 192.168.202.131@tcp [ 2384.984394] Key type .llcrypt unregistered [ 2384.986550] Key type ._llcrypt unregistered [ 2418.021044] Key type ._llcrypt registered [ 2418.022779] Key type .llcrypt registered [ 2418.200476] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2419.826748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2419.916209] alg: No test for adler32 (adler32-zlib) [ 2421.260584] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 2421.688342] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2423.488891] Key type lgssc registered [ 2425.283494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2436.388290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2436.421316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.104733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.269653] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug +malloc all [ 2448.370033] LustreError: 75923:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.419505] LustreError: 75923:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2453.165093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.481417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2453.659431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2459.365488] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug +malloc all [ 2468.468098] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2475.418911] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2483.522033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2483.705924] LustreError: 77745:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2483.958600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2489.331881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2491.871310] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug +malloc all [ 2499.286819] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2508.769284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2508.771848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.778783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2508.811802] Lustre: Skipped 1 previous similar message [ 2513.889889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2513.902138] Lustre: Skipped 1 previous similar message [ 2514.673842] Lustre: server umount lustre-OST0000 complete [ 2520.034932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2520.045479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2520.075436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2520.078204] Lustre: Skipped 1 previous similar message [ 2524.132600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.133725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2524.149161] Lustre: Skipped 1 previous similar message [ 2525.866544] Lustre: server umount lustre-MDT0000 complete [ 2529.248977] LustreError: 75919:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.267205] LustreError: 75919:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2529.749979] LustreError: 75904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862222 with bad export cookie 554427967793819645 [ 2529.751185] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2529.759795] LustreError: 75904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2530.154228] Lustre: server umount lustre-MDT0001 complete [ 2543.500266] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2547.612409] Key type lgssc unregistered [ 2548.053758] LNet: 79459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.063339] LNetError: 79459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.090366] LNet: Removed LNI 192.168.202.131@tcp [ 2548.901755] Key type .llcrypt unregistered [ 2548.904083] Key type ._llcrypt unregistered [ 2579.194144] Key type ._llcrypt registered [ 2579.195219] Key type .llcrypt registered [ 2579.250966] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2580.499246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2580.528968] alg: No test for adler32 (adler32-zlib) [ 2581.564556] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 2581.777678] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2583.464346] Key type lgssc registered [ 2584.245307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2596.930345] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2596.963094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2612.576606] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2617.696963] LustreError: 80852:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 2644.320576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.984453] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2660.865813] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2670.145778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2670.214308] Lustre: *** cfs_fail_loc=706, val=0*** [ 2670.488037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.553940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2675.463428] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2675.687751] LustreError: 80877:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.707322] LustreError: 80877:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2679.781551] LustreError: 81937:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.819335] LustreError: 81937:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2684.898340] LustreError: 81936:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.912911] LustreError: 81936:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2686.010285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.511885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2693.138824] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2702.448189] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2709.288141] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2716.641510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2716.646247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2716.661902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2717.173852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2717.176561] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2717.199223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2717.215924] Lustre: Skipped 1 previous similar message [ 2722.282018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2722.292060] Lustre: Skipped 1 previous similar message [ 2726.369742] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2726.535205] Lustre: server umount lustre-OST0000 complete [ 2732.515633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.531587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2732.543483] Lustre: Skipped 1 previous similar message [ 2732.565856] Lustre: Skipped 1 previous similar message [ 2737.634607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2737.644914] Lustre: Skipped 1 previous similar message [ 2737.755448] Lustre: server umount lustre-MDT0000 complete [ 2741.910744] LustreError: 80859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862434 with bad export cookie 7326292378430574973 [ 2741.921075] LustreError: 80859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2741.921589] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2742.354565] Lustre: server umount lustre-MDT0001 complete [ 2755.177884] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2757.790956] Key type lgssc unregistered [ 2758.086815] LNet: 84475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2758.091039] LNetError: 84475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2758.102339] LNet: Removed LNI 192.168.202.131@tcp [ 2758.942564] Key type .llcrypt unregistered [ 2758.945277] Key type ._llcrypt unregistered [ 2788.379437] Key type ._llcrypt registered [ 2788.382806] Key type .llcrypt registered [ 2788.562137] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2789.847189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2789.903305] alg: No test for adler32 (adler32-zlib) [ 2791.455877] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 2791.989961] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2793.728641] Key type lgssc registered [ 2795.590686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2805.553243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2805.571839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2812.411395] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2821.462979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.330072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2827.535308] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2827.751489] LustreError: 86581:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.782584] LustreError: 86581:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2832.865373] LustreError: 86585:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.908333] LustreError: 86585:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2836.703391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2837.075686] LustreError: 86581:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.097635] LustreError: 86581:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2837.118960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2842.596833] LustreError: 86581:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.614837] LustreError: 86581:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2844.869163] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2847.722531] LustreError: 87327:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.784976] LustreError: 87327:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2857.401886] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2857.431436] Lustre: MGS: Client d40b176d-b8a4-4e77-babf-bd871e227be3 (at 0@lo) reconnecting [ 2857.494384] LustreError: 86595:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2858.073782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.190174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2865.163662] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2883.553255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2883.554919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.567168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2883.579261] Lustre: Skipped 1 previous similar message [ 2883.593643] Lustre: Skipped 1 previous similar message [ 2886.965917] Lustre: server umount lustre-OST0000 complete [ 2893.809410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2893.821994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.837795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2893.848631] Lustre: Skipped 1 previous similar message [ 2893.865778] Lustre: Skipped 1 previous similar message [ 2897.540252] Lustre: server umount lustre-MDT0000 complete [ 2898.914404] LustreError: 88138:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.934092] LustreError: 88138:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2902.929970] Lustre: server umount lustre-MDT0001 complete [ 2907.453810] Lustre: server umount MGS complete [ 2921.164549] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:30:13 (1787862613) [ 2955.343623] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 2967.646149] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3013.649525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3019.803559] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3031.152212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.566662] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3031.626497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3031.744707] Lustre: lustre-MDT0001: new disk, initializing [ 3031.941080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3031.993223] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3032.012149] Lustre: 93778:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3032.018556] LustreError: 93778:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3037.667856] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3051.946157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3052.428452] Lustre: lustre-OST0000: new disk, initializing [ 3052.436119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3052.454144] Lustre: 94547:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3052.722045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.298496] Lustre: 94738:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008cea7a28 x1874709693113984/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787862768 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3061.477397] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3073.163335] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3079.653271] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3080.916661] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3080.943729] Lustre: MGS: Client f568dd12-a839-4036-b66e-6eb81660ff93 (at 0@lo) reconnecting [ 3080.978709] LustreError: 93764:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3081.022161] Lustre: 93045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3081.097390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3081.247087] Lustre: lustre-MDT0000: new disk, initializing [ 3081.504755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.557994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3082.658078] Lustre: 95617:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b2e064d8 x1874709693121408/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787862791 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3086.847484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3086.856723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3087.455064] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3087.750815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3087.920400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 3117.537761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3117.544640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.553036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3120.872056] Lustre: server umount lustre-OST0000 complete [ 3127.777987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3127.784511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3127.798944] Lustre: Skipped 1 previous similar message [ 3127.809294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3127.827555] Lustre: Skipped 2 previous similar messages [ 3132.028583] Lustre: server umount lustre-MDT0000 complete [ 3132.897071] LustreError: 93750:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.912903] LustreError: 93750:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3143.457505] Lustre: server umount lustre-MDT0001 complete [ 3147.110360] Lustre: server umount MGS complete [ 3159.366578] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:34:11 (1787862851) [ 3177.865731] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3180.823638] Key type lgssc unregistered [ 3181.104275] LNet: 98119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3181.110281] LNetError: 98119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3181.122425] LNet: Removed LNI 192.168.202.131@tcp [ 3181.894382] Key type .llcrypt unregistered [ 3181.896115] Key type ._llcrypt unregistered [ 3199.832767] Key type ._llcrypt registered [ 3199.834298] Key type .llcrypt registered [ 3200.010916] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3201.772403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3201.856778] alg: No test for adler32 (adler32-zlib) [ 3203.106559] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 3203.390971] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3205.080460] Key type lgssc registered [ 3206.729985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3216.033975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3216.047418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3217.508473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3222.944048] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3227.618570] LustreError: 99345:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3227.639926] LustreError: 99345:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3232.259301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.570754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3236.857637] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3246.638300] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3254.126444] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3262.887430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3263.487844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.505951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3271.026046] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3280.524904] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3294.181098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3294.190510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.201354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3294.252781] Lustre: Skipped 1 previous similar message [ 3299.321880] Lustre: server umount lustre-OST0000 complete [ 3309.542847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3309.548531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.551559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.551572] Lustre: Skipped 1 previous similar message [ 3309.619385] Lustre: Skipped 1 previous similar message [ 3311.679848] Lustre: server umount lustre-MDT0000 complete [ 3314.658305] LustreError: 99344:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.690852] LustreError: 99344:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3315.683219] LustreError: 100548:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3315.706732] LustreError: 100548:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3316.055112] LustreError: 99325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863009 with bad export cookie 15330005592572422932 [ 3316.060837] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3316.069639] LustreError: 99325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3317.070768] Lustre: server umount lustre-MDT0001 complete [ 3333.482971] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3336.853594] Key type lgssc unregistered [ 3337.108829] LNet: 102881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3337.118579] LNetError: 102881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3337.142842] LNet: Removed LNI 192.168.202.131@tcp [ 3338.026321] Key type .llcrypt unregistered [ 3338.028590] Key type ._llcrypt unregistered [ 3357.329346] Key type ._llcrypt registered [ 3357.330959] Key type .llcrypt registered [ 3357.480435] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3359.668645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3359.772468] alg: No test for adler32 (adler32-zlib) [ 3361.123319] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 3361.600400] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3363.368220] Key type lgssc registered [ 3364.872551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3372.123898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3372.135273] LustreError: 103964:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3372.169890] LustreError: 103968:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3372.178464] LustreError: 103964:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4973 [ 3372.187220] LustreError: 103964:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3372.197770] LustreError: 103964:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3372.202214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3374.003487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3383.017540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3383.295472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3389.435829] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3396.316319] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing lsmod [ 3399.657991] LustreError: 105175:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3399.661354] LustreError: 105174:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3399.673852] LustreError: 105175:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 3399.686603] LustreError: 105175:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3399.695362] LustreError: 105175:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3399.712738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3400.221986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.746680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3403.760886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.782830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.281954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.283049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3405.287585] Lustre: Skipped 1 previous similar message [ 3405.303914] Lustre: Skipped 1 previous similar message [ 3408.426706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3408.436616] Lustre: Skipped 1 previous similar message [ 3409.823354] Lustre: server umount lustre-MDT0000 complete [ 3413.475458] LustreError: 105103:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3413.505148] LustreError: 105103:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 3414.575192] LustreError: 103981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863107 with bad export cookie 10413055362220347745 [ 3414.584877] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3414.591591] LustreError: 103981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3414.609640] Lustre: 103998:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.131@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3414.688315] LustreError: 104437:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000af48ee1c x1874710289731456/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3414.981251] Lustre: server umount lustre-MDT0001 complete [ 3429.575229] Lustre: server umount lustre-OST0000 complete [ 3439.815950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3440.363067] LustreError: 106239:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3440.484588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.734710] LustreError: 106240:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3445.784064] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3450.853227] LustreError: 106239:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3455.463787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.981880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3461.316476] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3470.556450] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3477.014481] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3485.930844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3486.440995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3494.602400] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3503.671571] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3510.028057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3525.626585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.626625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3525.627864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3525.654335] Lustre: Skipped 2 previous similar messages [ 3525.699784] Lustre: Skipped 1 previous similar message [ 3528.143291] Lustre: server umount lustre-OST0000 complete [ 3532.775592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3532.785820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.801285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3532.805247] Lustre: Skipped 1 previous similar message [ 3539.148515] Lustre: server umount lustre-MDT0000 complete [ 3540.968279] LustreError: 108841:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3540.989064] LustreError: 108841:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3544.002624] LustreError: 106220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863237 with bad export cookie 10413055362220348438 [ 3544.019793] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3544.039408] LustreError: 106220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3544.556897] Lustre: server umount lustre-MDT0001 complete [ 3558.250747] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3562.092176] Key type lgssc unregistered [ 3562.439734] LNet: 109800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3562.448246] LNetError: 109800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3562.472799] LNet: Removed LNI 192.168.202.131@tcp [ 3563.803224] Key type .llcrypt unregistered [ 3563.807224] Key type ._llcrypt unregistered [ 3592.637144] Key type ._llcrypt registered [ 3592.638632] Key type .llcrypt registered [ 3592.883441] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3595.604677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3595.652450] alg: No test for adler32 (adler32-zlib) [ 3596.954722] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 3597.302344] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3599.040186] Key type lgssc registered [ 3600.791869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3610.951549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3610.980730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3612.682409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3617.776548] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3622.886994] LustreError: 111218:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3622.916597] LustreError: 111218:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3626.768948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.233117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3633.654499] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3643.315662] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3651.679253] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3659.184813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3659.804645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3665.749343] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3673.862621] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3682.311334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3690.132225] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3693.784686] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3702.755740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3702.781526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3702.816143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3704.577497] Lustre: server umount lustre-OST0000 complete [ 3709.409322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3709.415433] LustreError: Skipped 1 previous similar message [ 3709.418688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3709.426493] Lustre: Skipped 1 previous similar message [ 3709.432397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3709.437271] Lustre: Skipped 1 previous similar message [ 3712.998342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3713.002754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3715.204396] Lustre: server umount lustre-MDT0000 complete [ 3718.123556] LustreError: 111215:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3718.132208] LustreError: 111215:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3718.823972] LustreError: 111199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863411 with bad export cookie 11645784881133844750 [ 3718.832868] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3718.851414] LustreError: 111199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3719.255150] Lustre: server umount lustre-MDT0001 complete [ 3731.812821] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3734.665366] Key type lgssc unregistered [ 3734.970704] LNet: 114885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3734.975750] LNetError: 114885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3734.994314] LNet: Removed LNI 192.168.202.131@tcp [ 3735.883559] Key type .llcrypt unregistered [ 3735.886301] Key type ._llcrypt unregistered [ 3755.169905] Key type ._llcrypt registered [ 3755.171827] Key type .llcrypt registered [ 3755.278501] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3756.303873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3756.399407] alg: No test for adler32 (adler32-zlib) [ 3757.510499] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 3757.758961] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3759.473090] Key type lgssc registered [ 3760.885506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3769.217297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3769.238268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3770.697933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3775.701993] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3780.086613] LustreError: 116110:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3780.100705] LustreError: 116110:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3784.347976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.667583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3789.092141] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3795.991455] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3801.270036] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3807.602858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3808.028682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.682160] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3815.233610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3823.190511] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3834.851300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3834.870115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3834.884975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3835.876842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3835.877359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3835.877819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.884742] Lustre: Skipped 1 previous similar message [ 3837.354348] Lustre: server umount lustre-OST0000 complete [ 3846.115829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.125966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3846.141973] Lustre: Skipped 1 previous similar message [ 3847.477994] Lustre: server umount lustre-MDT0000 complete [ 3850.989987] LustreError: 116089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863544 with bad export cookie 13649036229773686819 [ 3850.997171] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3850.999258] LustreError: 116089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3851.493651] Lustre: server umount lustre-MDT0001 complete [ 3866.840178] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3870.286600] Key type lgssc unregistered [ 3870.743732] LNet: 119646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3870.754526] LNetError: 119646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3870.774873] LNet: Removed LNI 192.168.202.131@tcp [ 3871.635584] Key type .llcrypt unregistered [ 3871.637349] Key type ._llcrypt unregistered [ 3901.039628] Key type ._llcrypt registered [ 3901.043854] Key type .llcrypt registered [ 3901.159278] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3902.264333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3902.289809] alg: No test for adler32 (adler32-zlib) [ 3903.483897] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 3903.829547] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3905.508986] Key type lgssc registered [ 3906.693461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3915.145122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3915.170541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3916.786078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3921.525721] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3925.987261] LustreError: 121068:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3926.001623] LustreError: 121068:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3929.862451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.252055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3935.042335] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3942.751805] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3948.873116] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3955.874141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.207479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3961.346400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3963.406573] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3971.542539] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3982.448590] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3986.337597] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3986.342911] Lustre: Skipped 2 previous similar messages [ 3988.085204] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3988.107264] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3991.032934] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3991.039097] Lustre: Skipped 2 previous similar messages [ 3993.856796] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3993.860388] Lustre: Skipped 1 previous similar message [ 3998.327867] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.31@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3998.342225] Lustre: Skipped 2 previous similar messages [ 4002.522024] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.31@tcp 192.168.0.[2,10]@tcp [ 4005.903771] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.31/24@tcp in log lustre-MDT0000 [ 4005.911568] Lustre: Skipped 2 previous similar messages [ 4009.804405] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.31/24@tcp [ 4009.810200] Lustre: Skipped 1 previous similar message [ 4016.613933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4016.620170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.639148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4016.676195] Lustre: Skipped 1 previous similar message [ 4021.732896] LustreError: 122888:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.893097] Lustre: server umount lustre-OST0000 complete [ 4026.851569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4026.857450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4026.879546] Lustre: Skipped 1 previous similar message [ 4026.903669] Lustre: Skipped 2 previous similar messages [ 4031.973315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4031.981463] Lustre: Skipped 1 previous similar message [ 4032.476358] Lustre: server umount lustre-MDT0000 complete [ 4036.230096] LustreError: 124660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863729 with bad export cookie 4415022985209042656 [ 4036.232936] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4036.238653] LustreError: 124660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4036.844671] Lustre: server umount lustre-MDT0001 complete [ 4051.090901] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4055.574107] Key type lgssc unregistered [ 4056.071992] LNet: 125873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4056.079860] LNetError: 125873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4056.109740] LNet: Removed LNI 192.168.202.131@tcp [ 4057.361192] Key type .llcrypt unregistered [ 4057.368358] Key type ._llcrypt unregistered [ 4091.492561] Key type ._llcrypt registered [ 4091.494718] Key type .llcrypt registered [ 4091.627467] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4092.834423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4092.980281] alg: No test for adler32 (adler32-zlib) [ 4094.058289] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 4094.267533] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4095.922488] Key type lgssc registered [ 4097.248764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4105.346549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4105.369788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.825908] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4106.858608] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.31/24@tcp [ 4106.864685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4111.981628] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4116.968657] LustreError: 127437:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.005322] LustreError: 127437:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4120.866571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4121.110482] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4121.132994] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.31/24@tcp [ 4121.155335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4125.158536] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4133.931939] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4140.730505] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4149.351040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4149.908673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.002662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 4157.107817] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4168.018151] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4181.989915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.998136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4182.006072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4182.028915] Lustre: Skipped 1 previous similar message [ 4186.545590] Lustre: server umount lustre-OST0000 complete [ 4192.228943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.235412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4192.264135] Lustre: Skipped 1 previous similar message [ 4192.279697] Lustre: Skipped 2 previous similar messages [ 4197.346385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4197.351780] Lustre: Skipped 1 previous similar message [ 4198.319658] Lustre: server umount lustre-MDT0000 complete [ 4202.478969] LustreError: 128797:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.479986] LustreError: 127419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863895 with bad export cookie 6658593313527972597 [ 4202.517917] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4202.562510] LustreError: 127419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4203.062810] Lustre: server umount lustre-MDT0001 complete [ 4215.940877] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4218.565841] Key type lgssc unregistered [ 4218.873878] LNet: 131029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4218.880467] LNetError: 131029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4218.898525] LNet: Removed LNI 192.168.202.131@tcp [ 4219.971749] Key type .llcrypt unregistered [ 4219.973891] Key type ._llcrypt unregistered [ 4247.694320] Key type ._llcrypt registered [ 4247.699173] Key type .llcrypt registered [ 4247.808749] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 4264.058708] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4265.211771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4265.232560] alg: No test for adler32 (adler32-zlib) [ 4266.406673] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 4266.652280] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4268.321105] Key type lgssc registered [ 4269.740728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4300.847691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4300.858455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.159414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4302.197280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4302.322233] Lustre: lustre-MDT0000: new disk, initializing [ 4302.396119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4302.413820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4307.719257] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4319.708304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4319.833256] Lustre: 134433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4319.934906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4319.943460] Lustre: Skipped 1 previous similar message [ 4320.043577] Lustre: lustre-MDT0001: new disk, initializing [ 4320.148344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4320.223611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4320.250223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4325.561158] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4337.518609] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4343.709825] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4350.511268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4350.780961] Lustre: lustre-OST0000: new disk, initializing [ 4350.786692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4350.793317] Lustre: 136402:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4350.862599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4357.762708] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4358.264180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4358.285050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4358.481188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4368.030948] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4372.598666] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4372.795647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4378.142812] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4378.522664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4384.230441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.250136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4384.266293] Lustre: Skipped 1 previous similar message [ 4384.290574] Lustre: Skipped 1 previous similar message [ 4386.533985] Lustre: server umount lustre-OST0000 complete [ 4391.918913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4391.930358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4391.958560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4394.475872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4394.483145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.709940] Lustre: server umount lustre-MDT0000 complete [ 4402.365659] LustreError: 134424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864095 with bad export cookie 11987295848959812102 [ 4402.387406] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4402.401333] LustreError: 134424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4402.717747] Lustre: server umount lustre-MDT0001 complete [ 4412.195157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4412.766125] LustreError: 138620:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.788712] LustreError: 138620:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4412.841884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4418.018622] LustreError: 138621:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4418.679629] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4422.119202] LustreError: 138620:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.245110] LustreError: 138621:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4428.600802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4429.016025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4435.296676] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4443.279948] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4449.211686] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4456.157787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4456.473431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4462.781932] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4472.264401] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4480.623491] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4480.835123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4486.676431] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4486.943130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4495.793378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4496.008157] Lustre: lustre-OST0001: new disk, initializing [ 4496.018378] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4496.022823] Lustre: 141991:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4496.164626] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4501.498229] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4502.747365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4502.761654] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4502.964374] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4511.715216] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4511.961783] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4518.444500] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4518.792922] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4526.781083] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98d1894b8800.ost_server_uuid 50 [ 4528.597911] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98d1894b8800.ost_server_uuid in IDLE state after 0 sec [ 4538.850537] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4538.857641] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4538.875703] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4538.881744] Lustre: Skipped 1 previous similar message [ 4538.888188] Lustre: Skipped 2 previous similar messages [ 4543.994964] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4544.004385] Lustre: Skipped 1 previous similar message [ 4544.710475] Lustre: server umount lustre-OST0001 complete [ 4549.093715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4549.103081] LustreError: 140440:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4549.132982] LustreError: 140440:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4554.210260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4554.212987] Lustre: Skipped 3 previous similar messages [ 4554.873569] Lustre: server umount lustre-OST0000 complete [ 4559.332756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4559.344630] Lustre: Skipped 2 previous similar messages [ 4564.570112] Lustre: server umount lustre-MDT0000 complete [ 4569.442600] LustreError: 138601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864262 with bad export cookie 11987295848959812984 [ 4569.458511] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4569.472243] LustreError: 138601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4570.174656] Lustre: server umount lustre-MDT0001 complete [ 4591.396939] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4594.764811] Key type lgssc unregistered [ 4595.075941] LNet: 145001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4595.083652] LNetError: 145001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4595.099355] LNet: Removed LNI 192.168.202.131@tcp [ 4596.305807] Key type .llcrypt unregistered [ 4596.308699] Key type ._llcrypt unregistered [ 4644.936632] Key type ._llcrypt registered [ 4644.940876] Key type .llcrypt registered [ 4645.077447] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 4663.875577] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4665.324595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4665.365116] alg: No test for adler32 (adler32-zlib) [ 4666.480643] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 4666.832788] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4668.592880] Key type lgssc registered [ 4670.265591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4714.434626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4714.489325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4715.790779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4715.816930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4715.904846] Lustre: lustre-MDT0000: new disk, initializing [ 4715.994839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4716.022879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4721.227892] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4732.510279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4732.705935] Lustre: 149170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4732.780138] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4732.790536] Lustre: Skipped 1 previous similar message [ 4732.963001] Lustre: lustre-MDT0001: new disk, initializing [ 4733.173694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4733.262797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4733.280342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4739.284716] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4749.840924] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4756.528900] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4764.410499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4764.785956] Lustre: lustre-OST0000: new disk, initializing [ 4764.790615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4764.801046] Lustre: 151141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4764.950472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4772.594592] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4774.011045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4774.028554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4774.251595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4783.500823] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4789.002123] Lustre: Failing over lustre-OST0000 [ 4789.739484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4789.741663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4789.745638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4789.756789] Lustre: Skipped 1 previous similar message [ 4789.775082] Lustre: Skipped 1 previous similar message [ 4791.236146] Lustre: server umount lustre-OST0000 complete [ 4811.393727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4811.880268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.895776] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4818.316409] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4819.330547] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4823.682138] Lustre: Failing over lustre-MDT0000 [ 4824.079809] Lustre: server umount lustre-MDT0000 complete [ 4825.058898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4825.081714] Lustre: Skipped 2 previous similar messages [ 4833.645147] LustreError: 150185:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4833.657365] LustreError: 150185:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 4834.274813] LustreError: 150313:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4834.282706] LustreError: 150313:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4838.764477] LustreError: 149178:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4841.442626] Lustre: 147393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787864518/real 1787864518] req@00000000491a2d6c x1874711658151040/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787864534 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4841.480686] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4841.510493] LustreError: 150313:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4841.537103] LustreError: 150313:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 4843.997257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.563684] LustreError: 149181:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4846.612636] LustreError: 149181:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4852.009603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4852.020070] LustreError: Skipped 1 previous similar message [ 4852.067575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4852.098416] Lustre: lustre-OST0000: Recovery over after 0:34, of 3 clients 3 recovered and 0 were evicted. [ 4852.104562] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4852.172252] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4854.694191] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4857.337279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4857.343626] Lustre: Skipped 1 previous similar message [ 4857.361752] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4857.408286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4858.407135] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4867.554780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4867.559273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4867.602411] Lustre: Skipped 1 previous similar message [ 4867.627426] Lustre: Skipped 1 previous similar message [ 4872.680738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4872.690184] Lustre: Skipped 1 previous similar message [ 4873.149405] Lustre: server umount lustre-OST0000 complete [ 4877.796138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4877.796182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4877.797152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4877.815324] Lustre: Skipped 1 previous similar message [ 4877.838558] Lustre: Skipped 1 previous similar message [ 4882.914594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4882.922608] Lustre: Skipped 1 previous similar message [ 4891.104765] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4891.232894] Lustre: server umount lustre-MDT0000 complete [ 4893.157578] LustreError: 153767:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4893.166639] LustreError: 153767:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 4895.075677] LustreError: 149965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864588 with bad export cookie 3207319752961329367 [ 4895.095698] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4895.531850] Lustre: server umount lustre-MDT0001 complete [ 4907.305168] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4912.165861] Key type lgssc unregistered [ 4912.560521] LNet: 154813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4912.570433] LNetError: 154813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4912.600860] LNet: Removed LNI 192.168.202.131@tcp [ 4913.801504] Key type .llcrypt unregistered [ 4913.805181] Key type ._llcrypt unregistered [ 4956.344539] Key type ._llcrypt registered [ 4956.347911] Key type .llcrypt registered [ 4956.481446] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4957.884407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4958.000740] alg: No test for adler32 (adler32-zlib) [ 4959.266152] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 4959.676462] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4961.432312] Key type lgssc registered [ 4962.969925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4972.305400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4972.332444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4973.835291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4977.890951] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 4983.278940] LustreError: 156338:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4983.309580] LustreError: 156338:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4988.388625] LustreError: 156337:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4990.864934] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4993.509826] LustreError: 156338:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4995.583567] LustreError: 156366:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4995.591517] LustreError: 156366:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4995.599671] LustreError: 156366:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4995.978858] Lustre: server umount lustre-MDT0000 complete [ 5012.677781] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 5024.741137] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5065.705727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.054036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5066.109334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5066.207436] Lustre: lustre-MDT0000: new disk, initializing [ 5066.301846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5066.326807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5072.391263] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5085.252254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5085.374968] Lustre: 159671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5085.423707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5085.427405] Lustre: Skipped 1 previous similar message [ 5085.518404] Lustre: lustre-MDT0001: new disk, initializing [ 5085.637501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5085.672481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5085.685980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5090.706234] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5100.694437] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5106.132874] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5111.792174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5112.076831] Lustre: lustre-OST0000: new disk, initializing [ 5112.081279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5112.088875] Lustre: 161639:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5112.184986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5118.678459] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5121.621904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5121.642498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5121.777224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5128.642473] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5131.890581] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5133.626737] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 17:07:05 (1787864825) [ 5362.145162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5362.161099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5362.177246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5367.492326] Lustre: server umount lustre-MDT0000 complete [ 5367.779142] LustreError: 161106:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5367.790708] LustreError: 161106:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5370.723885] LustreError: 159663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865063 with bad export cookie 1792213602998676826 [ 5370.730154] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5370.732297] LustreError: 159663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5372.903250] LustreError: 159678:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5372.907077] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5372.948935] Lustre: Skipped 2 previous similar messages [ 5372.954454] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5372.963372] Lustre: Skipped 2 previous similar messages [ 5375.972381] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5380.069486] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5385.185082] LustreError: 161106:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5385.185170] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5385.187138] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 5385.210528] LustreError: 161106:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5385.629205] Lustre: server umount lustre-MDT0001 complete [ 5395.839885] Lustre: server umount lustre-OST0000 complete [ 5405.676625] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 5415.557697] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5449.790520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5449.956635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5449.982897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5450.063835] Lustre: lustre-MDT0000: new disk, initializing [ 5450.121311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5450.135029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5453.963586] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5463.760983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5463.912735] Lustre: 166299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5463.989662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5464.001135] Lustre: Skipped 1 previous similar message [ 5464.134844] Lustre: lustre-MDT0001: new disk, initializing [ 5464.280429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5464.364908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5464.377792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5469.498732] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5479.444930] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5485.343511] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5491.368567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5491.634484] Lustre: lustre-OST0000: new disk, initializing [ 5491.638502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5491.656942] Lustre: 168266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5491.756291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5497.943165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5497.960377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5498.110447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5498.316426] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5507.938533] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5512.291719] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5512.489433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5516.849942] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5517.034336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5523.940371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5523.940535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5523.952252] LustreError: Skipped 1 previous similar message [ 5523.954618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5523.977638] Lustre: Skipped 1 previous similar message [ 5533.152353] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5533.297802] Lustre: server umount lustre-OST0000 complete [ 5539.297803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5539.309908] Lustre: Skipped 1 previous similar message [ 5543.120109] Lustre: server umount lustre-MDT0000 complete [ 5544.423525] LustreError: 166309:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5544.433776] LustreError: 166309:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5546.581594] LustreError: 166290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865239 with bad export cookie 1792213602998835656 [ 5546.587144] LustreError: 166290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5546.589607] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5546.861922] Lustre: server umount lustre-MDT0001 complete [ 5554.223330] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:14:06 (1787865246) [ 5571.692560] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 5582.456763] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5621.222929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5621.540414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5621.573459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5621.665772] Lustre: lustre-MDT0000: new disk, initializing [ 5621.721126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5621.734597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5626.888278] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5637.813804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5637.902572] Lustre: 173027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5638.083175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5642.743668] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5651.599545] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5657.776335] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5665.690377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5665.976411] Lustre: lustre-OST0000: new disk, initializing [ 5665.978897] Lustre: Skipped 1 previous similar message [ 5665.981561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5665.987843] Lustre: Skipped 2 previous similar messages [ 5665.994879] Lustre: 175002:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5671.550344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5671.567088] Lustre: Skipped 1 previous similar message [ 5671.575715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5671.675953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5673.531152] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5684.664787] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5692.387627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5692.392370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5692.410398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5692.410408] Lustre: Skipped 5 previous similar messages [ 5698.783164] Lustre: server umount lustre-OST0000 complete [ 5704.684235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5704.702667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5704.743864] Lustre: Skipped 1 previous similar message [ 5710.486548] Lustre: server umount lustre-MDT0000 complete [ 5712.870371] LustreError: 173037:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5712.884846] LustreError: 173037:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5714.389563] LustreError: 173018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865407 with bad export cookie 1792213602998836538 [ 5714.395684] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5714.405832] LustreError: 173018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5714.882579] Lustre: server umount lustre-MDT0001 complete [ 5724.670695] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:16:56 (1787865416) [ 5741.018359] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 5751.506662] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5791.394170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5791.687851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5791.789765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5791.899045] Lustre: lustre-MDT0000: new disk, initializing [ 5792.093205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.108551] Lustre: Skipped 2 previous similar messages [ 5792.150464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5797.808771] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5808.841594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.952437] Lustre: 179322:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5809.142313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5813.610654] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5823.360942] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5830.483725] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5838.676520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5839.062609] Lustre: 181304:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5845.064521] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5845.215150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5846.420000] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5857.273238] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5865.960378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5865.962508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5865.970095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5865.970103] Lustre: Skipped 6 previous similar messages [ 5865.987140] Lustre: Skipped 2 previous similar messages [ 5866.001247] LustreError: Skipped 1 previous similar message [ 5870.888385] Lustre: server umount lustre-OST0000 complete [ 5875.697983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5881.091627] Lustre: server umount lustre-MDT0000 complete [ 5881.319799] LustreError: 179335:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5881.354696] LustreError: 179335:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5885.658882] LustreError: 181303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865578 with bad export cookie 1792213602998837616 [ 5885.671155] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5885.675752] LustreError: 181303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5886.184576] Lustre: server umount lustre-MDT0001 complete [ 5898.575946] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 5902.444633] Key type lgssc unregistered [ 5902.808131] LNet: 183160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5902.822414] LNetError: 183160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5902.863654] LNet: Removed LNI 192.168.202.131@tcp [ 5904.326711] Key type .llcrypt unregistered [ 5904.331920] Key type ._llcrypt unregistered [ 5929.349960] Key type ._llcrypt registered [ 5929.351638] Key type .llcrypt registered [ 5929.478968] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 5930.462440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5930.553337] alg: No test for adler32 (adler32-zlib) [ 5931.699187] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 5932.016909] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 5933.704373] Key type lgssc registered [ 5934.933892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5943.311283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5943.335875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.823743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5950.286307] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5954.025588] LustreError: 184576:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5954.035687] LustreError: 184576:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5958.860089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.296364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5964.065531] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 5972.415954] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5978.997648] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5986.846366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5987.311888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5989.405126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5994.539337] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6002.611227] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6025.189928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6025.191177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6025.219909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6025.234803] Lustre: Skipped 1 previous similar message [ 6028.356892] Lustre: server umount lustre-OST0000 complete [ 6035.427109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6035.428870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6035.442298] Lustre: Skipped 2 previous similar messages [ 6035.448221] Lustre: Skipped 1 previous similar message [ 6038.679634] Lustre: server umount lustre-MDT0000 complete [ 6040.545591] LustreError: 185586:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6040.559624] LustreError: 185586:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6043.072369] LustreError: 184558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865736 with bad export cookie 4302169615614603416 [ 6043.079529] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6043.094266] LustreError: 184558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6043.480629] Lustre: server umount lustre-MDT0001 complete [ 6054.880239] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 6057.755798] Key type lgssc unregistered [ 6058.089597] LNet: 188138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6058.095832] LNetError: 188138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6059.110060] LNet: Removed LNI 192.168.202.131@tcp [ 6060.258733] Key type .llcrypt unregistered [ 6060.262793] Key type ._llcrypt unregistered [ 6089.711891] Key type ._llcrypt registered [ 6089.714443] Key type .llcrypt registered [ 6089.857405] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 6091.107890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6091.246026] alg: No test for adler32 (adler32-zlib) [ 6092.479410] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 6092.725657] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 6094.425116] Key type lgssc registered [ 6095.707461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6102.915252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6102.929712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6104.323451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.445990] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6114.792037] LustreError: 189557:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6114.816507] LustreError: 189557:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6116.480412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6116.713725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6122.526643] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6129.702221] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6133.944233] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6139.506937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6140.000875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6147.794860] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6149.247447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6155.863675] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6163.853310] Lustre: lustre-OST0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 6164.963214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6164.973804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.992506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6165.001945] Lustre: Skipped 1 previous similar message [ 6168.991328] Lustre: server umount lustre-OST0000 complete [ 6176.753130] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff98d183df3800.ost_server_uuid 50 [ 6178.409902] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6193.634788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6193.655451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6193.671066] Lustre: Skipped 1 previous similar message [ 6193.676139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6195.681749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6195.683915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.359540] Lustre: server umount lustre-MDT0000 complete [ 6200.806102] LustreError: 190803:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6200.838943] LustreError: 190803:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6203.608987] LustreError: 189536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865896 with bad export cookie 10795913360237773391 [ 6203.618569] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6203.620024] LustreError: 189536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6204.039783] Lustre: server umount lustre-MDT0001 complete [ 6213.146346] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:25:05 (1787865905) [ 6219.792153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6220.154400] LustreError: 193471:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6220.216824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6224.414699] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6225.380404] LustreError: 193472:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6230.499828] LustreError: 193471:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.292670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6233.796653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6238.728159] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6246.847876] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6252.461909] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6259.750932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6260.182558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.334131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6267.239730] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6274.129956] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6281.066949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6281.463866] Lustre: lustre-OST0001: new disk, initializing [ 6281.470431] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6281.478640] Lustre: 196310:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6281.629369] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6290.250540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6290.259755] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6290.508380] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6290.869277] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6304.932752] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6309.346755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6309.369493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6309.411425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6309.424566] Lustre: Skipped 1 previous similar message [ 6314.782634] Lustre: server umount lustre-OST0000 complete [ 6316.003163] LustreError: 196643:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.047769] LustreError: 196643:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6323.387973] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6323.846515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6329.511606] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6329.767302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6334.449912] LustreError: 196643:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6334.467494] LustreError: 196643:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 6344.678415] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6344.702706] Lustre: Skipped 1 previous similar message [ 6344.708128] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6344.723067] Lustre: Skipped 4 previous similar messages [ 6350.350132] Lustre: server umount lustre-OST0001 complete [ 6354.914443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6354.923412] Lustre: Skipped 1 previous similar message [ 6361.053149] Lustre: server umount lustre-MDT0000 complete [ 6366.243386] LustreError: 193453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866059 with bad export cookie 10795913360237774462 [ 6366.258023] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6366.263294] LustreError: 193453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6366.912369] Lustre: server umount lustre-MDT0001 complete [ 6395.937862] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:28:08 (1787866088) [ 6404.573198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6404.994828] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6405.060368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6405.473549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6411.136856] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6422.812522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6422.916613] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6422.975293] Lustre: 199754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6429.916976] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6438.552937] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6444.026455] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6451.090055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6451.276749] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6457.806917] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6461.625310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6465.543083] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6472.114388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6472.224760] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6472.469552] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6472.483803] Lustre: Skipped 2 previous similar messages [ 6479.602872] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6481.607911] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6487.470739] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6492.131148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6492.147917] Lustre: Skipped 1 previous similar message [ 6492.152655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6492.155293] Lustre: Skipped 6 previous similar messages [ 6495.809665] Lustre: server umount lustre-OST0000 complete [ 6497.257463] LustreError: 202145:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6497.273890] LustreError: 202145:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 6512.623034] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6518.287567] Lustre: server umount lustre-OST0001 complete [ 6525.923911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6525.939216] LustreError: Skipped 1 previous similar message [ 6525.942981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6525.969247] Lustre: Skipped 3 previous similar messages [ 6525.990263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.009175] Lustre: Skipped 5 previous similar messages [ 6529.481996] Lustre: server umount lustre-MDT0000 complete [ 6533.044637] LustreError: 199746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866226 with bad export cookie 10795913360237775869 [ 6533.049971] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6533.054392] LustreError: 199746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6533.362655] Lustre: server umount lustre-MDT0001 complete [ 6563.035421] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:30:55 (1787866255) [ 6579.155576] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 6587.695710] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 6621.057835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6621.201717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6621.216748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6621.260121] Lustre: lustre-MDT0000: new disk, initializing [ 6621.320492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6621.330384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6624.939856] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6634.788225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6634.893583] Lustre: 207854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6634.942084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6634.945397] Lustre: Skipped 1 previous similar message [ 6635.005877] Lustre: lustre-MDT0001: new disk, initializing [ 6635.137607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6635.153442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6638.414627] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6647.567261] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6653.952318] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6660.710914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.049295] Lustre: lustre-OST0000: new disk, initializing [ 6661.051940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6661.060606] Lustre: 209823:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6667.368823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6667.384423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6667.523972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6669.086699] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6679.060802] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6684.425416] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6684.580692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6689.178813] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6689.358048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6693.348446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6693.360537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6693.367400] Lustre: Skipped 1 previous similar message [ 6693.369544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6693.369552] Lustre: Skipped 2 previous similar messages [ 6693.390090] LustreError: Skipped 1 previous similar message [ 6697.754407] Lustre: server umount lustre-OST0000 complete [ 6708.059021] Lustre: server umount lustre-MDT0000 complete [ 6708.705898] LustreError: 207866:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6708.731752] LustreError: 207866:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 6711.665477] LustreError: 207847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866404 with bad export cookie 10795913360237777654 [ 6711.668784] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6711.676855] LustreError: 207847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6712.000897] Lustre: server umount lustre-MDT0001 complete [ 6720.654901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6725.818160] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6735.258490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.452462] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6749.483615] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6756.151137] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6762.826389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6770.012271] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6776.909332] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6781.489756] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6781.722056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6785.549445] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6785.761384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6791.138210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6791.144725] LustreError: Skipped 1 previous similar message [ 6793.848461] Lustre: server umount lustre-OST0000 complete [ 6800.718835] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6805.301173] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6834.225945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6839.893329] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6847.288207] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6852.999753] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6856.040264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6856.060200] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6856.072710] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6861.288619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6861.291921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6861.293924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6861.293932] Lustre: Skipped 6 previous similar messages [ 6861.302133] Lustre: Skipped 6 previous similar messages [ 6861.311071] LustreError: Skipped 1 previous similar message [ 6862.558815] Lustre: server umount lustre-OST0000 complete [ 6872.838796] Lustre: server umount lustre-MDT0000 complete [ 6876.647211] LustreError: 213193:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6876.663239] LustreError: 213193:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 6877.013818] LustreError: 212024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866570 with bad export cookie 10795913360237778529 [ 6877.020938] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6877.026174] LustreError: 212024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6877.491168] Lustre: server umount lustre-MDT0001 complete [ 6885.643554] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:36:18 (1787866578) [ 6892.719831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6893.212959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6893.218522] Lustre: Skipped 6 previous similar messages [ 6897.848490] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6907.299528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6912.236395] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6919.393975] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6925.005710] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6932.412230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6939.757994] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6947.586732] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6952.754073] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6952.996516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6958.062319] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6958.289431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6966.001362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6966.226330] Lustre: lustre-OST0001: new disk, initializing [ 6966.230862] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6966.237703] Lustre: 221379:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6972.590138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6972.604937] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6972.800084] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6974.279455] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 6984.059686] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6989.300342] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6989.531389] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6994.267830] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6994.702336] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6998.522963] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6998.528040] LustreError: Skipped 1 previous similar message [ 7003.519442] Lustre: server umount lustre-OST0001 complete [ 7011.349452] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7011.575796] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7015.839664] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7016.201216] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7044.411262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7049.720318] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7049.746813] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7049.748610] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7049.767259] Lustre: Skipped 2 previous similar messages [ 7050.715978] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7057.494912] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7059.076672] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7075.297697] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7075.548606] Lustre: server umount lustre-OST0001 complete [ 7099.038467] Lustre: server umount lustre-OST0000 complete [ 7109.928106] Lustre: server umount lustre-MDT0000 complete [ 7113.955607] LustreError: 217991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866806 with bad export cookie 10795913360237779502 [ 7113.956922] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7113.966404] LustreError: 217991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7120.336653] Lustre: server umount lustre-MDT0001 complete [ 7142.535656] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:40:34 (1787866834) [ 7149.431808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7149.659926] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7149.687925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7153.873518] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7161.514792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7161.642297] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7161.677107] Lustre: 226621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7167.007778] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7176.835423] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7183.318924] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7191.753158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7191.935133] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7199.453301] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7207.341486] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7217.727929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7217.844063] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7226.856585] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7236.382958] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7238.180448] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7243.686516] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7243.882934] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7249.254411] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7249.513651] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7255.756438] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98d186a3f800.ost_server_uuid 50 [ 7257.627520] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98d186a3f800.ost_server_uuid in IDLE state after 0 sec [ 7258.682771] Lustre: Permanently deactivating lustre-OST0001 [ 7258.696431] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7262.199413] Lustre: Permanently reactivating lustre-OST0001 [ 7267.820336] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7267.823176] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7267.836048] LustreError: Skipped 3 previous similar messages [ 7267.854073] Lustre: Skipped 10 previous similar messages [ 7267.861707] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7267.870575] Lustre: Skipped 20 previous similar messages [ 7272.548939] Lustre: server umount lustre-OST0001 complete [ 7272.941118] LustreError: 228468:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7272.953840] LustreError: 228468:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 37 previous similar messages [ 7282.690864] Lustre: server umount lustre-OST0000 complete [ 7292.951263] Lustre: server umount lustre-MDT0000 complete [ 7296.506202] LustreError: 226613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866989 with bad export cookie 10795913360237780790 [ 7296.507876] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7296.519365] LustreError: 226613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7296.917962] Lustre: server umount lustre-MDT0001 complete [ 7320.924567] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:43:33 (1787867013) [ 7327.973295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.313180] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7328.359447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7328.367285] Lustre: Skipped 5 previous similar messages [ 7331.962183] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7340.055294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7340.140388] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7340.224760] Lustre: 233315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7345.065520] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7353.742027] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7359.402484] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7366.676517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7366.974136] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7367.022565] Lustre: Permanently deactivating lustre-OST0000 [ 7367.041840] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7372.813885] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7373.542578] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7380.884391] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7387.713608] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7387.782647] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7394.455798] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7397.017360] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7402.370249] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7404.079726] Lustre: Permanently reactivating lustre-OST0000 [ 7404.089349] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7404.093236] Lustre: Skipped 2 previous similar messages [ 7410.830598] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7410.832660] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 38 seconds. I think it's dead, and I am evicting it. exp 000000008d010248, cur 1787867103 deadline 1787867095 last 1787867065 [ 7410.857660] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7410.940732] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7411.053558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7436.257288] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7436.490487] Lustre: server umount lustre-OST0001 complete [ 7455.200144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7455.500359] Lustre: server umount lustre-OST0000 complete [ 7466.242437] Lustre: server umount lustre-MDT0000 complete [ 7470.189939] LustreError: 236912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867163 with bad export cookie 10795913360237782463 [ 7470.208185] LustreError: 236912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7470.210304] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7470.519892] Lustre: server umount lustre-MDT0001 complete [ 7481.733505] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 7484.851765] Key type lgssc unregistered [ 7485.139966] LNet: 238205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7485.146110] LNetError: 238205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7485.175225] LNet: Removed LNI 192.168.202.131@tcp [ 7486.157464] Key type .llcrypt unregistered [ 7486.167458] Key type ._llcrypt unregistered [ 7510.193528] Key type ._llcrypt registered [ 7510.195236] Key type .llcrypt registered [ 7510.355684] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 7511.832738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7512.055116] alg: No test for adler32 (adler32-zlib) [ 7513.424699] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 7513.815646] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 7515.688146] Key type lgssc registered [ 7517.330223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7528.435903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7528.461941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7529.832927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7534.566767] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7539.177790] LustreError: 239681:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7539.198723] LustreError: 239681:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7543.872134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7543.980281] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7543.996714] Lustre: 239667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7544.001562] Lustre: Permanently deactivating lustre-MDT0001 [ 7544.007279] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7544.227687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7544.518555] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7544.520897] LustreError: 239711:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7544.526443] LustreError: 239711:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7544.531833] LustreError: 239711:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7548.535642] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7556.236137] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7563.486141] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7569.574549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7569.788254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7570.797308] LustreError: 241523:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7570.843040] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7574.889102] LustreError: 241524:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7574.920693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7576.693362] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7580.135402] LustreError: 241795:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7580.193388] LustreError: 241795:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7583.709186] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7585.257620] LustreError: 241522:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7585.269646] LustreError: 241522:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7591.758308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7591.935910] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7597.044870] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7597.052070] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7598.499424] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7606.060242] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7609.330170] Lustre: Permanently reactivating lustre-MDT0001 [ 7609.350225] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7609.357871] Lustre: Skipped 2 previous similar messages [ 7616.957124] Lustre: lustre-MDT0001: Client f5390018-3267-427a-8e7d-aa6538ddddea (at 192.168.202.31@tcp) reconnecting [ 7625.628145] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7627.312312] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7633.843720] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7634.076379] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7640.025600] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7640.323696] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7642.438298] Lustre: Permanently deactivating lustre-MDT0001 [ 7642.460592] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7642.466477] Lustre: Skipped 2 previous similar messages [ 7648.888991] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7656.932392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7656.937216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7656.955325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7658.470417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7658.480179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7658.491698] Lustre: Skipped 2 previous similar messages [ 7658.519584] Lustre: Skipped 3 previous similar messages [ 7658.947218] Lustre: server umount lustre-MDT0000 complete [ 7662.928649] LustreError: 239662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867355 with bad export cookie 15099570376392241130 [ 7662.954794] LustreError: 239662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7662.960648] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7663.455886] Lustre: server umount lustre-MDT0001 complete [ 7667.269581] Lustre: server umount lustre-OST0000 complete [ 7677.751529] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:49:30 (1787867370) [ 7689.780485] Lustre: server umount lustre-OST0001 complete [ 7699.075679] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 7712.212861] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 7749.474359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7749.864536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7749.873726] Lustre: Skipped 2 previous similar messages [ 7749.946949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7750.131415] Lustre: lustre-MDT0000: new disk, initializing [ 7750.337795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7750.355318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7756.465422] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7767.110749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7767.216242] Lustre: 247937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7767.262201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7767.269859] Lustre: Skipped 1 previous similar message [ 7767.374784] Lustre: lustre-MDT0001: new disk, initializing [ 7767.494387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7767.536804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7767.551637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7772.632645] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7783.044985] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7788.535522] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7795.591160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7795.936493] Lustre: lustre-OST0000: new disk, initializing [ 7795.943891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7795.949860] Lustre: 249907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7796.048420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7802.175345] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7804.978427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7804.989285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7805.069807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7812.813863] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7815.283330] LustreError: 249567:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7825.175025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7825.386752] LustreError: 249567:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7825.407834] Lustre: lustre-OST0001: new disk, initializing [ 7825.414281] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7825.417586] LustreError: 247943:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7825.422927] Lustre: 251020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7832.056810] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7834.760570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7834.763317] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7834.822846] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7835.472414] LustreError: 247943:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7842.338895] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7844.034033] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7847.328367] Lustre: lustre-OST0001: Not available for connect from 192.168.202.31@tcp (stopping) [ 7850.466108] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7850.466471] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7850.480441] LustreError: Skipped 1 previous similar message [ 7850.487915] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7850.494478] Lustre: Skipped 1 previous similar message [ 7850.513119] Lustre: Skipped 1 previous similar message [ 7852.790896] Lustre: server umount lustre-OST0001 complete [ 7855.603145] LustreError: 249890:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7855.621717] LustreError: 249890:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 7870.951184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.956396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7870.972510] Lustre: Skipped 1 previous similar message [ 7870.976216] Lustre: Skipped 1 previous similar message [ 7876.073742] LustreError: 251497:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7876.086744] LustreError: 251497:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 7876.723478] Lustre: server umount lustre-OST0000 complete [ 7881.187357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7881.211122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7881.218498] Lustre: Skipped 3 previous similar messages [ 7887.294319] Lustre: server umount lustre-MDT0000 complete [ 7891.341254] LustreError: 247931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867584 with bad export cookie 15099570376392243944 [ 7891.342806] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7891.350679] LustreError: 247931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7897.785459] Lustre: server umount lustre-MDT0001 complete [ 7909.554455] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 7913.218566] Key type lgssc unregistered [ 7913.640460] LNet: 253059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7913.650952] LNetError: 253059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7913.667333] LNet: Removed LNI 192.168.202.131@tcp [ 7914.817646] Key type .llcrypt unregistered [ 7914.820842] Key type ._llcrypt unregistered [ 7960.601859] Key type ._llcrypt registered [ 7960.604324] Key type .llcrypt registered [ 7960.743133] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 7962.160149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7962.342923] alg: No test for adler32 (adler32-zlib) [ 7963.803271] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 7964.201638] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 7965.984807] Key type lgssc registered [ 7967.556941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7975.886891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7975.918422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7977.294764] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7977.355475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7977.670790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7982.950077] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 7992.445603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7992.583684] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7992.609925] Lustre: 255272:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7992.794614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7997.639349] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8005.702272] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8011.361608] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8018.294896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8018.544819] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8018.750626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8025.166936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8025.936181] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8033.802181] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8091.621237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8091.629849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8091.648055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8091.668621] Lustre: Skipped 1 previous similar message [ 8095.895272] Lustre: server umount lustre-OST0000 complete [ 8100.819383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8114.726295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8115.301416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8122.321803] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8131.092168] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8131.692820] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8131.723220] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8131.729583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8131.759447] Lustre: Skipped 1 previous similar message [ 8173.030086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8173.033963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8173.050475] LustreError: Skipped 1 previous similar message [ 8173.066642] Lustre: Skipped 1 previous similar message [ 8173.078075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8173.096094] Lustre: Skipped 1 previous similar message [ 8176.970815] Lustre: server umount lustre-OST0000 complete [ 8182.244495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8182.260971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8182.296910] Lustre: Skipped 1 previous similar message [ 8182.333041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8188.188408] Lustre: server umount lustre-MDT0000 complete [ 8192.385474] LustreError: 255264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867885 with bad export cookie 7454809712750575254 [ 8192.394481] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8192.398817] LustreError: 255264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8192.961037] Lustre: server umount lustre-MDT0001 complete [ 8205.334405] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 8208.193670] Key type lgssc unregistered [ 8208.465823] LNet: 260698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8208.475061] LNetError: 260698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8209.512963] LNet: Removed LNI 192.168.202.131@tcp [ 8210.494591] Key type .llcrypt unregistered [ 8210.496789] Key type ._llcrypt unregistered [ 8237.161478] Key type ._llcrypt registered [ 8237.162550] Key type .llcrypt registered [ 8237.221834] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 8238.425599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8238.436777] alg: No test for adler32 (adler32-zlib) [ 8239.484644] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 8239.846457] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 8241.504174] Key type lgssc registered [ 8242.313835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8249.959276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8249.980777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8251.411465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8256.741840] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8261.610498] LustreError: 262118:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8261.626521] LustreError: 262118:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8266.330943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8266.807223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8272.162576] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8280.854836] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8288.095183] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8296.145497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8296.416247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8297.468039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8303.244159] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8313.419246] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8343.528096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8343.542775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.568357] Lustre: Skipped 1 previous similar message [ 8343.571972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8346.821788] Lustre: server umount lustre-OST0000 complete [ 8353.762200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.774636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8353.786147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8353.798527] LustreError: Skipped 1 previous similar message [ 8353.825140] Lustre: Skipped 2 previous similar messages [ 8357.779235] Lustre: server umount lustre-MDT0000 complete [ 8358.881254] LustreError: 262118:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8358.910426] LustreError: 262118:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8361.373314] LustreError: 262100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868054 with bad export cookie 12321497591725228772 [ 8361.386260] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8361.393122] LustreError: 262100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8361.892835] Lustre: server umount lustre-MDT0001 complete [ 8373.833902] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 8376.953226] Key type lgssc unregistered [ 8377.260806] LNet: 266438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8377.276462] LNetError: 266438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8377.312734] LNet: Removed LNI 192.168.202.131@tcp [ 8378.487558] Key type .llcrypt unregistered [ 8378.494823] Key type ._llcrypt unregistered [ 8398.090374] Key type ._llcrypt registered [ 8398.092379] Key type .llcrypt registered [ 8398.243562] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 8399.848601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8399.876039] alg: No test for adler32 (adler32-zlib) [ 8400.910726] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 8401.177335] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 8402.824583] Key type lgssc registered [ 8404.024995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8412.661787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8412.709739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8414.366194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8419.412182] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8424.939981] LustreError: 267662:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8424.966086] LustreError: 267662:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8429.118644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8429.558945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8434.837932] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8442.971437] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8449.750776] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8457.873910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8458.356691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8459.388982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8465.444393] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8473.111248] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8485.345310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8485.352132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8485.357821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8485.372480] Lustre: Skipped 1 previous similar message [ 8485.387046] Lustre: Skipped 1 previous similar message [ 8490.389441] Lustre: server umount lustre-OST0000 complete [ 8495.589733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8495.625658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8500.064618] Lustre: server umount lustre-MDT0000 complete [ 8500.705135] LustreError: 267658:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8500.736330] LustreError: 267658:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8503.978694] LustreError: 267644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868197 with bad export cookie 11711387577293921629 [ 8503.983663] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8503.994509] LustreError: 267644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8504.362686] Lustre: server umount lustre-MDT0001 complete [ 8517.914801] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 8521.252902] Key type lgssc unregistered [ 8521.602295] LNet: 271345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8521.616639] LNetError: 271345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8521.630899] LNet: Removed LNI 192.168.202.131@tcp [ 8522.612482] Key type .llcrypt unregistered [ 8522.617445] Key type ._llcrypt unregistered [ 8548.435645] Key type ._llcrypt registered [ 8548.437836] Key type .llcrypt registered [ 8548.547465] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 8549.688608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8549.837226] alg: No test for adler32 (adler32-zlib) [ 8550.951236] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 8551.222852] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 8553.002598] Key type lgssc registered [ 8554.572940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8563.897229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8563.924114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8565.464494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8570.325323] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8575.972807] LustreError: 272765:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8576.000280] LustreError: 272765:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8579.071048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8579.481536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8584.768253] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8593.622872] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8600.113954] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8608.230550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8608.725139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8611.844464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8615.796161] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8625.257517] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8652.772869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8652.786644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8652.818190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8654.830567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8654.831781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8654.870149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8658.171895] Lustre: server umount lustre-OST0000 complete [ 8665.058150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8665.067749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8665.072212] Lustre: Skipped 2 previous similar messages [ 8670.177598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8670.181295] Lustre: Skipped 1 previous similar message [ 8674.275577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8674.282973] Lustre: Skipped 1 previous similar message [ 8677.346539] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8677.637649] Lustre: server umount lustre-MDT0000 complete [ 8679.397646] LustreError: 273492:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8679.416686] LustreError: 273492:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8681.286209] LustreError: 272746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868374 with bad export cookie 18366623940444350184 [ 8681.308170] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8681.701370] Lustre: server umount lustre-MDT0001 complete [ 8694.192764] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 8697.451679] Key type lgssc unregistered [ 8697.729482] LNet: 277074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8697.733359] LNetError: 277074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8697.756413] LNet: Removed LNI 192.168.202.131@tcp [ 8698.630408] Key type .llcrypt unregistered [ 8698.633394] Key type ._llcrypt unregistered [ 8718.378212] Key type ._llcrypt registered [ 8718.385739] Key type .llcrypt registered [ 8718.502261] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 8720.037798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8720.101995] alg: No test for adler32 (adler32-zlib) [ 8721.641860] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 8722.026537] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 8723.760259] Key type lgssc registered [ 8725.938318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8736.856610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8736.870304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8738.712708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8744.773265] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8748.018110] LustreError: 278312:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8748.035277] LustreError: 278312:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8753.136284] LustreError: 278314:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8754.193878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8754.658310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8759.933928] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8769.127580] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8776.474574] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8784.373336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8784.725118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8793.333750] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8802.169859] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8808.320593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8823.786519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8823.791959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8823.794241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8823.798900] Lustre: Skipped 1 previous similar message [ 8828.707786] Lustre: server umount lustre-OST0000 complete [ 8834.027990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8834.029127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8834.046936] Lustre: Skipped 1 previous similar message [ 8834.055686] Lustre: Skipped 2 previous similar messages [ 8839.140475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8840.174305] Lustre: server umount lustre-MDT0000 complete [ 8843.313832] LustreError: 278304:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8843.334550] LustreError: 278304:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8844.053098] LustreError: 278277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868537 with bad export cookie 6629809522055723731 [ 8844.057116] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8844.071813] LustreError: 278277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8845.072879] Lustre: server umount lustre-MDT0001 complete [ 8858.509597] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 8861.655404] Key type lgssc unregistered [ 8862.274844] LNet: 282042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8862.296104] LNetError: 282042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8862.329973] LNet: Removed LNI 192.168.202.131@tcp [ 8863.632257] Key type .llcrypt unregistered [ 8863.641381] Key type ._llcrypt unregistered [ 8876.385598] Key type ._llcrypt registered [ 8876.387584] Key type .llcrypt registered [ 8876.536523] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8899.723656] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 8920.311341] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 8921.927106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8922.016141] alg: No test for adler32 (adler32-zlib) [ 8923.157113] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 8923.532766] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 8925.280232] Key type lgssc registered [ 8926.798741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8965.536626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8965.596427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8966.837973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8966.900599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8967.010338] Lustre: lustre-MDT0000: new disk, initializing [ 8967.095160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8967.115581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8972.183388] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8983.159583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8983.335689] Lustre: 285601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8983.406712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8983.412902] Lustre: Skipped 1 previous similar message [ 8983.479777] Lustre: lustre-MDT0001: new disk, initializing [ 8983.580249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8983.624151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8983.636432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8988.169131] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 8997.040396] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9002.224274] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9010.321140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9010.626202] Lustre: lustre-OST0000: new disk, initializing [ 9010.641735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9010.665969] Lustre: 287569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9010.840942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9017.536664] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9018.455972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9018.466543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9018.606044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9027.083102] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9031.943680] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9032.124965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9036.556835] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9036.801223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9039.329342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9039.345993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9039.352052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9039.362302] Lustre: Skipped 1 previous similar message [ 9039.376235] Lustre: Skipped 1 previous similar message [ 9044.455189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9044.468470] Lustre: Skipped 1 previous similar message [ 9045.585952] Lustre: server umount lustre-OST0000 complete [ 9049.573800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9049.586793] Lustre: Skipped 1 previous similar message [ 9049.588263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9049.603042] Lustre: Skipped 1 previous similar message [ 9054.691957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9054.715684] Lustre: Skipped 1 previous similar message [ 9055.904821] Lustre: server umount lustre-MDT0000 complete [ 9059.839110] LustreError: 285592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868752 with bad export cookie 4896131226796537733 [ 9059.847428] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9059.861474] LustreError: 285592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9060.184819] Lustre: server umount lustre-MDT0001 complete [ 9069.571953] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 18:12:41 (1787868761) [ 9076.610158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9077.326084] LustreError: 289981:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9077.342446] LustreError: 289981:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9077.459611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9082.856777] LustreError: 289982:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9082.987196] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9087.974156] LustreError: 289981:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.989981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.091383] LustreError: 289982:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9093.304970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9099.233932] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9107.944598] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9115.192782] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9122.930896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9123.332856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9130.992402] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9139.469376] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9154.021815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9154.044690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9154.047254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9154.050027] LustreError: Skipped 1 previous similar message [ 9155.257465] Lustre: server umount lustre-OST0000 complete [ 9159.144627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9159.175150] Lustre: Skipped 2 previous similar messages [ 9164.260724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9164.268652] Lustre: Skipped 3 previous similar messages [ 9165.691335] Lustre: server umount lustre-MDT0000 complete [ 9169.380929] LustreError: 289978:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9169.409328] LustreError: 289978:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9170.060806] LustreError: 289963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868863 with bad export cookie 4896131226796538615 [ 9170.066830] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9170.083138] LustreError: 289963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9170.645883] Lustre: server umount lustre-MDT0001 complete [ 9184.759608] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 9188.300257] Key type lgssc unregistered [ 9188.550861] LNet: 293524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9188.555890] LNetError: 293524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9188.572689] LNet: Removed LNI 192.168.202.131@tcp [ 9189.315151] Key type .llcrypt unregistered [ 9189.316813] Key type ._llcrypt unregistered [ 9218.038616] Key type ._llcrypt registered [ 9218.040576] Key type .llcrypt registered [ 9218.131206] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 9233.186576] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 9234.141408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9234.309108] alg: No test for adler32 (adler32-zlib) [ 9235.311366] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 9235.482140] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 9237.120355] Key type lgssc registered [ 9238.044639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9293.195921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9293.276216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9294.719787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9294.770572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9294.952721] Lustre: lustre-MDT0000: new disk, initializing [ 9295.188088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9295.231868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9300.558424] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9312.397311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9312.575836] Lustre: 297350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9312.654351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9312.663364] Lustre: Skipped 1 previous similar message [ 9312.750910] Lustre: lustre-MDT0001: new disk, initializing [ 9312.891989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9312.949731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9312.960909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9318.881373] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9329.598575] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9336.353706] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9344.992707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9345.508197] Lustre: lustre-OST2710: new disk, initializing [ 9345.528809] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9345.550430] Lustre: 299323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9345.705801] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9353.214778] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9353.287561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9353.298067] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9353.433192] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9364.003269] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9372.924981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9373.232222] Lustre: lustre-OST03e8: new disk, initializing [ 9373.235756] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9373.255746] Lustre: 300388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9373.424675] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9378.405452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9378.414221] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9378.565190] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9381.429807] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9389.885686] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9398.220660] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9398.526276] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9403.596559] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9403.796931] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9409.296582] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9409.634410] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9415.679456] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9415.946689] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9424.881568] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9424.888845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9424.911300] Lustre: Skipped 3 previous similar messages [ 9424.919027] Lustre: Skipped 2 previous similar messages [ 9428.831528] Lustre: server umount lustre-MDT0000 complete [ 9432.967116] LustreError: 298156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869126 with bad export cookie 12939199375247112799 [ 9432.982354] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9432.993503] LustreError: 298156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9433.354741] Lustre: server umount lustre-MDT0001 complete [ 9447.404805] Lustre: server umount lustre-OST2710 complete [ 9451.297131] Lustre: 295149:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787869128/real 1787869128] req@000000006bcef6ac x1874716449120256/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787869144 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9451.323038] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9451.987150] Lustre: server umount lustre-OST03e8 complete [ 9472.283863] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 9484.587107] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 9527.322913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9527.491872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9527.511859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9527.558664] Lustre: lustre-MDT0000: new disk, initializing [ 9527.605533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9527.615973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9533.227512] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9544.819822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9545.000526] Lustre: 305639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9545.136423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9545.144468] Lustre: Skipped 1 previous similar message [ 9545.277641] Lustre: lustre-MDT0001: new disk, initializing [ 9545.394813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9545.443178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9545.456588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9551.034296] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9562.793350] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9569.966734] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9577.751245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9577.986827] Lustre: lustre-OST0000: new disk, initializing [ 9577.991827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9577.998276] Lustre: 307611:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9578.076748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9584.736477] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9585.270333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9585.281693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9585.423684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9595.562920] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9600.830596] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9601.134988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9606.654750] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9606.935718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9611.236180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9611.237782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9611.248762] LustreError: Skipped 1 previous similar message [ 9611.250406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9611.258151] Lustre: Skipped 1 previous similar message [ 9611.277290] Lustre: Skipped 1 previous similar message [ 9615.583952] Lustre: server umount lustre-OST0000 complete [ 9621.486405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9621.491270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9621.500850] Lustre: Skipped 1 previous similar message [ 9621.526928] Lustre: Skipped 1 previous similar message [ 9626.597242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9631.716358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9631.727124] Lustre: Skipped 2 previous similar messages [ 9634.274418] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9634.541771] Lustre: server umount lustre-MDT0000 complete [ 9636.834657] LustreError: 306729:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9636.857119] LustreError: 306729:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 9638.914236] LustreError: 305631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869331 with bad export cookie 12939199375247114493 [ 9638.926053] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9638.926355] LustreError: 305631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9639.255946] Lustre: server umount lustre-MDT0001 complete [ 9648.972674] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:22:20 (1787869340) [ 9651.014436] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9652.919958] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:22:24 (1787869344) [ 9657.304367] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 9690.871943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9691.248868] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9691.285390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9691.524040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9696.563777] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9706.009378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9706.287517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9706.327414] Lustre: 311473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9712.625118] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9735.919407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9736.175330] Lustre: Denying initial registration attempt from nid 192.168.202.131@tcp, specified as failover [ 9736.190096] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9736.201105] LustreError: 312925:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9736.207166] LustreError: 312925:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9736.209719] LustreError: 312925:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9736.309705] Lustre: server umount lustre-OST0000 complete [ 9736.316320] LustreError: 312925:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9747.938141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9747.951484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9747.956288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9750.385255] Lustre: server umount lustre-MDT0000 complete [ 9753.060471] LustreError: 311479:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9753.085625] LustreError: 311479:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9755.174494] LustreError: 311464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869448 with bad export cookie 12939199375247115375 [ 9755.187637] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9755.615743] Lustre: server umount lustre-MDT0001 complete [ 9770.577925] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 9774.174234] Key type lgssc unregistered [ 9774.528191] LNet: 314050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9774.543724] LNetError: 314050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9774.572864] LNet: Removed LNI 192.168.202.131@tcp [ 9775.642862] Key type .llcrypt unregistered [ 9775.645598] Key type ._llcrypt unregistered [ 9789.071554] Key type ._llcrypt registered [ 9789.074546] Key type .llcrypt registered [ 9789.172805] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 9790.604634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9790.672541] alg: No test for adler32 (adler32-zlib) [ 9791.798760] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 9792.107844] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 9793.800756] Key type lgssc registered [ 9795.067895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9843.788334] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 9851.874622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9851.900317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9853.175962] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9853.189424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9853.319180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9859.552216] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9869.160550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9869.405442] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9869.510994] Lustre: 316830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9869.789143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9875.807888] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9885.432546] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9892.012679] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9912.881237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9913.176112] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9913.304651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9919.512620] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 9927.460505] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9931.245940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9931.257835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9931.257975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9931.277329] Lustre: Skipped 1 previous similar message [ 9931.302073] Lustre: Skipped 1 previous similar message [ 9936.384296] Lustre: server umount lustre-OST0000 complete [ 9941.473923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9941.483120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9941.496213] Lustre: Skipped 1 previous similar message [ 9941.512797] Lustre: Skipped 1 previous similar message [ 9946.827906] Lustre: server umount lustre-MDT0000 complete [ 9950.889261] LustreError: 316821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869643 with bad export cookie 9951255235729074541 [ 9950.890162] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9950.905104] LustreError: 316821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9951.495636] Lustre: server umount lustre-MDT0001 complete [ 9964.121749] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 9967.690877] Key type lgssc unregistered [ 9968.133624] LNet: 320408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9968.146525] LNetError: 320408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9968.163188] LNet: Removed LNI 192.168.202.131@tcp [ 9969.522851] Key type .llcrypt unregistered [ 9969.525841] Key type ._llcrypt unregistered [ 9998.707076] Key type ._llcrypt registered [ 9998.708589] Key type .llcrypt registered [ 9998.802877] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 9999.831836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9999.859351] alg: No test for adler32 (adler32-zlib) [10000.958328] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [10001.142348] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [10002.819102] Key type lgssc registered [10004.503601] Lustre: Echo OBD driver; http://www.lustre.org/ [10015.439059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10015.463348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10017.069451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10021.996811] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10027.490225] LustreError: 321828:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10027.499700] LustreError: 321828:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10029.939464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10030.225192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10035.440475] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10043.401294] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10048.199396] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10057.962736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10058.389146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10064.966387] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10073.271160] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10081.763583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10081.769329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10081.780655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10086.311467] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [10086.370540] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10088.204350] Lustre: server umount lustre-MDT0000 complete [10091.382311] LustreError: 323028:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10092.066633] LustreError: 321807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869785 with bad export cookie 18017336258286420685 [10092.075534] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10092.076283] LustreError: 321807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10096.492850] LustreError: 324321:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10096.520538] LustreError: 324321:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10096.530593] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.31@tcp (stopping) [10096.544429] Lustre: Skipped 3 previous similar messages [10096.613085] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10096.625985] Lustre: Skipped 1 previous similar message [10098.474657] Lustre: server umount lustre-MDT0001 complete [10105.108141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10114.856831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10117.088288] LustreError: 325410:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10117.091568] LustreError: 321136:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e02308db x1874717252296576/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10117.582716] LustreError: 323661:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10117.615382] LustreError: 323661:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10117.754653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10117.839868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10122.725328] LustreError: 323662:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10123.585723] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10127.865444] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10132.962843] LustreError: 325422:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10132.985939] LustreError: 325422:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10133.278824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10133.723113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10138.992264] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10139.115180] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10146.725834] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10148.144917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10153.323806] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10155.007352] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10163.688136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10163.695194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10163.719495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10163.731311] Lustre: Skipped 2 previous similar messages [10164.722977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10166.447921] Lustre: server umount lustre-OST0000 complete [10174.955399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10174.963423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10174.973042] Lustre: Skipped 1 previous similar message [10174.974037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10175.016026] Lustre: Skipped 3 previous similar messages [10176.626824] Lustre: server umount lustre-MDT0000 complete [10179.995601] LustreError: 321808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869873 with bad export cookie 18017336258286428273 [10180.017196] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10180.027764] LustreError: 321808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10180.273351] Lustre: server umount lustre-MDT0001 complete [10192.369732] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [10195.478946] Key type lgssc unregistered [10195.847815] LNet: 328098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10195.851194] LNetError: 328098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10195.861054] LNet: Removed LNI 192.168.202.131@tcp [10196.605493] Key type .llcrypt unregistered [10196.607857] Key type ._llcrypt unregistered [10224.584131] Key type ._llcrypt registered [10224.585593] Key type .llcrypt registered [10224.729559] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [10226.308882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10226.329052] alg: No test for adler32 (adler32-zlib) [10227.568842] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [10227.958643] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [10229.689630] Key type lgssc registered [10231.087686] Lustre: Echo OBD driver; http://www.lustre.org/ [10240.067973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10240.086324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10241.723565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10246.727192] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10251.237477] LustreError: 329520:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10251.256157] LustreError: 329520:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10254.520431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10254.933519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10259.261843] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10266.314767] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10271.884532] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10275.301813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10275.313238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10275.333158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10276.324460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10276.326300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10279.460178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10279.472255] Lustre: Skipped 2 previous similar messages [10281.354199] Lustre: server umount lustre-MDT0000 complete [10284.514665] LustreError: 329514:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10284.540108] LustreError: 329514:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10284.656640] LustreError: 329501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869977 with bad export cookie 1039710715011722485 [10284.666607] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10284.666874] LustreError: 329501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10284.943310] Lustre: server umount lustre-MDT0001 complete [10299.381951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10299.555509] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10299.576965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10299.735085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10304.495736] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10313.619844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10313.822937] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10313.856032] Lustre: 332107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10314.232580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10318.497451] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10325.771217] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10331.369739] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10342.645195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10342.976606] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10342.993326] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10343.002925] LustreError: 333992:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10343.014876] LustreError: 333992:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10343.027338] LustreError: 333992:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10343.175067] Lustre: server umount lustre-OST0000 complete [10343.177137] LustreError: 333992:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10354.695725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10354.930538] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10355.095819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10362.185256] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10364.647808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10370.797034] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10377.953765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10378.015088] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10378.085596] Lustre: lustre-OST0001: new disk, initializing [10378.093227] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10378.097635] Lustre: 335536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10378.164427] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10384.005817] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10387.632835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10387.641706] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10387.737889] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10391.687659] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10398.185465] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10398.190238] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10398.205047] Lustre: Skipped 1 previous similar message [10398.205367] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10398.251758] LustreError: Skipped 1 previous similar message [10401.246315] Lustre: server umount lustre-OST0001 complete [10403.312587] LustreError: 334459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10403.336030] LustreError: 334459:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10405.857739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10405.861143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10405.877373] LustreError: Skipped 1 previous similar message [10405.879312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10405.893577] Lustre: Skipped 1 previous similar message [10405.910413] Lustre: Skipped 2 previous similar messages [10408.421827] LustreError: 334461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10408.456396] LustreError: 334461:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10411.971594] Lustre: server umount lustre-OST0000 complete [10418.667356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10418.677324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10418.700568] Lustre: Skipped 1 previous similar message [10418.723087] Lustre: Skipped 3 previous similar messages [10423.407848] Lustre: server umount lustre-MDT0000 complete [10423.778763] LustreError: 333168:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10423.800244] LustreError: 333168:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10427.248349] LustreError: 334475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870120 with bad export cookie 1039710715011723038 [10427.250892] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10427.263508] LustreError: 334475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10427.654741] Lustre: server umount lustre-MDT0001 complete [10439.424625] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [10442.321219] Key type lgssc unregistered [10442.669638] LNet: 337497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10442.701408] LNetError: 337497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10443.761428] LNet: Removed LNI 192.168.202.131@tcp [10444.810246] Key type .llcrypt unregistered [10444.812352] Key type ._llcrypt unregistered [10523.076732] Key type ._llcrypt registered [10523.078496] Key type .llcrypt registered [10523.168450] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [10538.087155] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [10538.960247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10538.979356] alg: No test for adler32 (adler32-zlib) [10539.884953] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [10540.051783] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [10541.690596] Key type lgssc registered [10542.453755] Lustre: Echo OBD driver; http://www.lustre.org/ [10582.414725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10582.491152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10583.863355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10583.951433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10584.061415] Lustre: lustre-MDT0000: new disk, initializing [10584.145207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10584.163924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10589.180527] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10600.212536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10600.347910] Lustre: 342437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10600.397513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10600.401303] Lustre: Skipped 1 previous similar message [10600.483397] Lustre: lustre-MDT0001: new disk, initializing [10600.583991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10600.631125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10600.644739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10604.965421] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10614.746553] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10620.937377] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10628.042593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10628.369423] Lustre: lustre-OST0000: new disk, initializing [10628.372555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10628.384357] Lustre: 344406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10628.461374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10635.157852] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10635.393849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10635.407402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10635.614149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10645.523570] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10650.938136] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10651.152582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10656.150545] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10656.362609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10661.354226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10661.365029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10661.387481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10664.402456] Lustre: server umount lustre-OST0000 complete [10671.592892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10671.601027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10671.603506] Lustre: Skipped 2 previous similar messages [10671.617854] Lustre: Skipped 2 previous similar messages [10675.167541] Lustre: server umount lustre-MDT0000 complete [10679.283702] LustreError: 342429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870372 with bad export cookie 4246840141858480975 [10679.288223] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10679.298244] LustreError: 342429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10679.776161] Lustre: server umount lustre-MDT0001 complete [10689.498422] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 18:39:41 (1787870381) [10698.079811] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 18:39:50 (1787870390) [10706.918668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10707.317542] LustreError: 347159:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10707.337081] LustreError: 347159:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10707.392527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10711.499195] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10712.546460] LustreError: 347160:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10717.671083] LustreError: 347159:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10718.962458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10719.280641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10723.288289] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10731.280764] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10738.580523] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10746.077728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10746.472403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10752.416398] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10760.384393] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10773.004641] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10775.129257] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10776.744568] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10778.888429] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10780.562799] Lustre: Failing over lustre-MDT0000 [10780.642680] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10780.663715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10780.680586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10780.699527] Lustre: Skipped 2 previous similar messages [10780.882275] Lustre: server umount lustre-MDT0000 complete [10782.614385] LustreError: 347156:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.701344] LustreError: 347155:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.750774] LustreError: 347155:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10796.007134] LustreError: 347154:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10796.046758] LustreError: 347154:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [10800.937789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10801.077308] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10801.320644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10801.364726] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10804.080363] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10806.500526] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10806.776461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10806.849268] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10806.944675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10816.298725] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10817.709822] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10820.970792] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10822.836638] Lustre: Failing over lustre-MDT0000 [10823.121928] Lustre: server umount lustre-MDT0000 complete [10824.583751] LustreError: 347155:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10824.606673] LustreError: 347155:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10827.235490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10827.243989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10827.262425] Lustre: Skipped 3 previous similar messages [10842.842586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10842.995410] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10843.214469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10843.262598] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10846.061566] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10848.245658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10848.253220] Lustre: Skipped 2 previous similar messages [10848.313632] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10848.384286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10848.876850] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10859.131675] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10860.951262] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10868.708154] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10868.710775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10868.726284] Lustre: Skipped 3 previous similar messages [10868.728894] Lustre: Skipped 1 previous similar message [10872.712683] Lustre: server umount lustre-MDT0000 complete [10873.836150] LustreError: 347155:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10873.869043] LustreError: 347155:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 18 previous similar messages [10876.785753] LustreError: 348999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870569 with bad export cookie 4246840141858483684 [10876.788330] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10876.795904] LustreError: 348999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10877.398399] Lustre: server umount lustre-MDT0001 complete [10892.889833] Lustre: server umount lustre-OST0000 complete [10911.419661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10911.953195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10915.597465] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10923.990958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10929.095667] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10937.863320] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10943.182243] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10950.486904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10956.282055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10958.056800] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [10966.755298] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10979.821525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10979.826073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10979.843577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10979.849025] Lustre: Skipped 1 previous similar message [10979.869783] Lustre: Skipped 2 previous similar messages [10983.311798] Lustre: server umount lustre-OST0000 complete [10990.052291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10993.763903] Lustre: server umount lustre-MDT0000 complete [10995.170507] LustreError: 353517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10995.185279] LustreError: 353517:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10998.388668] LustreError: 353501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870691 with bad export cookie 4246840141858484433 [10998.389428] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10998.413719] LustreError: 353501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10998.706394] Lustre: server umount lustre-MDT0001 complete [11011.593400] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [11015.322333] Key type lgssc unregistered [11015.683734] LNet: 357058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11015.689262] LNetError: 357058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11015.714903] LNet: Removed LNI 192.168.202.131@tcp [11016.621307] Key type .llcrypt unregistered [11016.622791] Key type ._llcrypt unregistered [11049.083887] Key type ._llcrypt registered [11049.088186] Key type .llcrypt registered [11049.237595] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [11050.430492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11050.519903] alg: No test for adler32 (adler32-zlib) [11051.597320] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [11051.905906] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [11053.592191] Key type lgssc registered [11054.799673] Lustre: Echo OBD driver; http://www.lustre.org/ [11062.538713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11062.564714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11063.938306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11069.833853] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11073.514756] LustreError: 358575:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11073.561685] LustreError: 358575:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11078.628020] LustreError: 358574:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11081.474791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11082.049103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11087.443619] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11096.027153] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11102.313968] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11108.795245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11109.042724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11115.927985] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11123.744880] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11133.638118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11143.649829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11143.656068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11143.668911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11144.162962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11144.175040] Lustre: Skipped 1 previous similar message [11144.185934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11144.190360] Lustre: Skipped 2 previous similar messages [11149.283390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11149.290345] Lustre: Skipped 2 previous similar messages [11149.617161] Lustre: server umount lustre-MDT0000 complete [11152.527444] LustreError: 358555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870845 with bad export cookie 16708070600663198812 [11152.538686] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11152.989377] Lustre: server umount lustre-MDT0001 complete [11167.028162] Lustre: server umount lustre-OST0000 complete [11179.158796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11179.471809] LustreError: 362312:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11179.511836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11184.633567] LustreError: 362313:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11184.666576] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11189.738331] LustreError: 362312:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11193.445495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11193.945964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11199.591797] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11208.468750] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11215.883285] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11226.574729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11227.091127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11234.255211] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11242.477816] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11249.580259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11265.506063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11265.511530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11265.528994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11270.113907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11270.114761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11270.139604] Lustre: Skipped 1 previous similar message [11270.160559] Lustre: Skipped 2 previous similar messages [11271.464599] Lustre: server umount lustre-MDT0000 complete [11275.239085] LustreError: 362308:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11275.263136] LustreError: 362308:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11275.290237] LustreError: 362294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870968 with bad export cookie 16708070600663199925 [11275.315863] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11275.632495] Lustre: server umount lustre-MDT0001 complete [11289.963053] Lustre: server umount lustre-OST0000 complete [11305.999537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11306.398662] LustreError: 366245:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11306.438247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11310.831833] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11318.487438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11322.524287] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11331.151413] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11337.356940] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11344.851653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11345.406652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11345.410588] Lustre: Skipped 1 previous similar message [11351.552408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11353.108583] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11360.849785] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11366.884429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11366.898706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11366.927083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11367.907996] Lustre: server umount lustre-OST0000 complete [11384.294486] Lustre: 357885:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787871061/real 1787871061] req@000000000ec6f534 x1874718353361024/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787871077 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11384.296160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11384.299271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11384.299281] Lustre: Skipped 4 previous similar messages [11384.320161] Lustre: 357885:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11384.340359] Lustre: Skipped 2 previous similar messages [11386.339479] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11386.568169] Lustre: server umount lustre-MDT0000 complete [11389.411560] LustreError: 366240:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11389.446138] LustreError: 366240:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11390.435695] Lustre: 357886:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787871067/real 1787871067] req@00000000d86ba427 x1874718353362944/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787871083 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11390.476461] Lustre: 357886:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11390.857225] LustreError: 366226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871083 with bad export cookie 16708070600663200954 [11390.873825] LustreError: 366975:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [11390.880064] LustreError: 366226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11390.881886] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11390.913478] LustreError: 366975:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [11391.278838] Lustre: server umount lustre-MDT0001 complete [11404.668910] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [11408.805242] Key type lgssc unregistered [11409.277633] LNet: 369782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11409.285184] LNetError: 369782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11409.312015] LNet: Removed LNI 192.168.202.131@tcp [11410.300246] Key type .llcrypt unregistered [11410.302148] Key type ._llcrypt unregistered [11438.826077] Key type ._llcrypt registered [11438.827246] Key type .llcrypt registered [11438.901848] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [11440.521772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11440.610192] alg: No test for adler32 (adler32-zlib) [11441.659348] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [11442.073330] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [11443.744867] Key type lgssc registered [11445.119880] Lustre: Echo OBD driver; http://www.lustre.org/ [11453.122403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11453.157991] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11453.175125] LustreError: 371227:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11453.221918] LustreError: 371227:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11453.234060] LustreError: 371227:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11453.257507] LustreError: 371227:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11453.272272] LustreError: 371227:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11462.428802] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11462.437052] LustreError: 371625:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11462.469233] LustreError: 371625:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11462.480449] LustreError: 371625:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11462.490484] LustreError: 371625:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11462.498791] LustreError: 371625:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11486.446195] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [11489.880636] Key type lgssc unregistered [11490.374419] LNet: 372632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11490.384299] LNetError: 372632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11490.412607] LNet: Removed LNI 192.168.202.131@tcp [11491.522492] Key type .llcrypt unregistered [11491.532502] Key type ._llcrypt unregistered [11513.950680] Key type ._llcrypt registered [11513.954493] Key type .llcrypt registered [11514.060915] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [11532.400480] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [11534.057223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11534.097280] alg: No test for adler32 (adler32-zlib) [11535.389420] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [11535.726775] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [11537.464206] Key type lgssc registered [11539.024638] Lustre: Echo OBD driver; http://www.lustre.org/ [11579.471620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11579.508193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11580.831453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11580.889219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11580.974689] Lustre: lustre-MDT0000: new disk, initializing [11581.037949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11581.050787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11586.173471] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11598.504939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11598.688561] Lustre: 375793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11598.761716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11598.764674] Lustre: Skipped 1 previous similar message [11598.936710] Lustre: lustre-MDT0001: new disk, initializing [11599.126641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11599.179887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11599.189705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11605.016870] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11615.428722] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11622.299730] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11629.017601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11629.153681] Lustre: lustre-OST0000: new disk, initializing [11629.156953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11629.170224] Lustre: 377760:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11629.308083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11636.928916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11636.960585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11637.220681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11637.442942] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11650.129798] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11656.159680] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11656.490344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11661.622348] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11661.853246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11667.944169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11667.945946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11667.969298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11667.983859] LustreError: Skipped 1 previous similar message [11670.422737] Lustre: server umount lustre-OST0000 complete [11676.129264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11676.132620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11676.141815] Lustre: Skipped 1 previous similar message [11676.147088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11676.153258] Lustre: Skipped 1 previous similar message [11678.180791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11678.183702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11678.199440] Lustre: Skipped 1 previous similar message [11681.344489] Lustre: server umount lustre-MDT0000 complete [11685.279115] LustreError: 375782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871378 with bad export cookie 14383242633553700164 [11685.293156] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11685.294389] LustreError: 375782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11685.789223] Lustre: server umount lustre-MDT0001 complete [11695.017266] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11697.030792] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:56:29 (1787871389) [11705.125492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11705.746804] LustreError: 380221:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11705.769462] LustreError: 380221:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11705.861311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11710.952344] LustreError: 380222:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11711.564939] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11715.051745] LustreError: 380221:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11720.162922] LustreError: 380222:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11721.087058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11721.496124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11726.586914] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11735.152159] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11740.365572] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11747.314622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11747.712459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11754.978716] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11764.224193] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11772.912133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11773.145391] Lustre: lustre-OST0001: new disk, initializing [11773.148614] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11773.153657] Lustre: 383058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11773.241068] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11779.278738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11779.291951] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11779.452916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11780.491395] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11792.013296] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11798.927944] Lustre: lustre-OST0001: Not available for connect from 192.168.202.31@tcp (stopping) [11800.041954] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11800.049763] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11800.058117] Lustre: Skipped 1 previous similar message [11802.243733] Lustre: server umount lustre-OST0001 complete [11804.020331] LustreError: 383367:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11815.408355] LustreError: 382044:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11815.422146] LustreError: 382044:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11822.562732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11822.574711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11822.581192] LustreError: Skipped 1 previous similar message [11822.583334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11822.583341] Lustre: Skipped 2 previous similar messages [11822.642981] Lustre: Skipped 1 previous similar message [11827.510306] Lustre: server umount lustre-OST0000 complete [11834.338425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11834.347362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11834.369880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11834.372741] Lustre: Skipped 3 previous similar messages [11847.136517] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11847.411310] Lustre: server umount lustre-MDT0000 complete [11850.214692] LustreError: 380222:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11850.232485] LustreError: 380222:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11851.689228] LustreError: 383069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871544 with bad export cookie 14383242633553701046 [11851.691645] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11851.701495] LustreError: 383069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11858.276891] Lustre: server umount lustre-MDT0001 complete [11869.884030] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [11873.932412] Key type lgssc unregistered [11874.204864] LNet: 385055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11874.210855] LNetError: 385055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11874.229740] LNet: Removed LNI 192.168.202.131@tcp [11875.126090] Key type .llcrypt unregistered [11875.127225] Key type ._llcrypt unregistered [11933.328764] Key type ._llcrypt registered [11933.330164] Key type .llcrypt registered [11933.419501] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [11935.186757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11935.256537] alg: No test for adler32 (adler32-zlib) [11936.386733] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [11936.713946] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [11938.432206] Key type lgssc registered [11939.728277] Lustre: Echo OBD driver; http://www.lustre.org/ [11947.950682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11947.965730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11949.308369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11949.358194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11949.639087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11955.335928] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11965.145244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11965.249894] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11965.294948] Lustre: 387861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11965.608691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11970.848264] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [11980.271684] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11987.173625] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11991.009290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11991.012782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11991.033997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11991.538321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11991.567265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11996.642881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11996.664131] Lustre: Skipped 1 previous similar message [12001.765176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12001.771390] Lustre: Skipped 1 previous similar message [12003.307965] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12003.834402] Lustre: server umount lustre-MDT0000 complete [12007.641776] LustreError: 387852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871700 with bad export cookie 7086719715448572464 [12007.659428] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12008.216738] Lustre: server umount lustre-MDT0001 complete [12019.283048] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:01:50 (1787871710) [12028.123639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12028.637243] LustreError: 390383:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12028.653499] LustreError: 390383:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12028.690096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12034.024660] LustreError: 390384:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12035.357040] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12039.146762] LustreError: 390383:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12044.258981] LustreError: 390384:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12045.343568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12045.888954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12052.181230] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12062.512883] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12069.145134] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12076.564372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12076.747284] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12076.967518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12084.680890] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12093.707922] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12102.505095] Lustre: Permanently deactivating lustre-OST0000 [12102.530349] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12112.921336] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12115.122149] LustreError: 393419:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12115.136363] LustreError: 393419:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12115.145538] LustreError: 393419:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12119.049758] LustreError: 393468:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12119.055815] LustreError: 393468:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12127.596203] Lustre: server umount lustre-OST0000 complete [12131.368116] LustreError: 393718:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12131.386799] LustreError: 393718:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12135.908493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12135.912611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12135.937041] Lustre: Skipped 1 previous similar message [12144.191093] Lustre: server umount lustre-MDT0000 complete [12146.154490] LustreError: 390383:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12146.185903] LustreError: 390383:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12148.535892] LustreError: 392218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871841 with bad export cookie 7086719715448573045 [12148.544840] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12148.553713] LustreError: 392218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12159.185592] Lustre: server umount lustre-MDT0001 complete [12168.367042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12173.458669] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12176.704496] LustreError: 394878:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12184.669635] Lustre: 395217:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.131@tcp [12187.185977] Lustre: server umount MGS complete [12201.942909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12202.523394] LustreError: 395942:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12202.559607] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12202.571057] Lustre: Skipped 1 previous similar message [12202.696623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12208.351146] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12218.241140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12218.785104] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12218.891163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12224.183298] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12232.292911] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12239.269552] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12241.566776] Lustre: Permanently reactivating lustre-OST0000 [12241.585845] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12241.593473] Lustre: Skipped 2 previous similar messages [12270.049308] LustreError: 397723:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12277.252147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12277.701316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12285.137860] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12295.946967] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12305.379071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12305.398328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12305.425673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12305.429623] Lustre: Skipped 3 previous similar messages [12306.401874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12309.595906] Lustre: server umount lustre-OST0000 complete [12316.129383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12316.138556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12316.154133] Lustre: Skipped 1 previous similar message [12321.003832] Lustre: server umount lustre-MDT0000 complete [12321.761380] LustreError: 395942:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12321.787307] LustreError: 395942:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12325.713150] LustreError: 395922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872018 with bad export cookie 7086719715448574095 [12325.714020] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12325.718842] LustreError: 395922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12326.270188] Lustre: server umount lustre-MDT0001 complete [12340.664214] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [12343.402563] Key type lgssc unregistered [12343.839116] LNet: 400314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12343.842781] LNetError: 400314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12344.871278] LNet: Removed LNI 192.168.202.131@tcp [12346.287167] Key type .llcrypt unregistered [12346.288449] Key type ._llcrypt unregistered [12367.188566] Key type ._llcrypt registered [12367.190514] Key type .llcrypt registered [12367.325581] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [12385.126799] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [12386.436371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12386.490036] alg: No test for adler32 (adler32-zlib) [12388.087422] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [12388.373838] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [12390.024794] Key type lgssc registered [12391.382447] Lustre: Echo OBD driver; http://www.lustre.org/ [12430.158751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12430.229269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12431.528776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12431.582304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12431.662794] Lustre: lustre-MDT0000: new disk, initializing [12431.755318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12431.772806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12437.198600] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12448.351125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12448.480738] Lustre: 403475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12448.525877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12448.529190] Lustre: Skipped 1 previous similar message [12448.587215] Lustre: lustre-MDT0001: new disk, initializing [12448.665367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12448.704682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12448.722850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12453.740834] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12464.708061] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12470.094538] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12476.518881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12476.876814] Lustre: lustre-OST0000: new disk, initializing [12476.881623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12476.888235] Lustre: 405447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12476.995227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12484.532149] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12486.260789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12486.278271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12486.451361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12494.277680] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12499.600816] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12499.853833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12504.618136] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12504.830905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12507.107352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12507.111648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12507.117204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12507.137601] Lustre: Skipped 1 previous similar message [12507.151603] Lustre: Skipped 1 previous similar message [12512.226337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12512.232319] Lustre: Skipped 1 previous similar message [12512.774182] Lustre: server umount lustre-OST0000 complete [12517.349160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12517.363463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12517.384605] Lustre: Skipped 1 previous similar message [12517.397624] Lustre: Skipped 1 previous similar message [12522.466464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12527.586657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12527.600148] Lustre: Skipped 2 previous similar messages [12531.168444] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12531.495431] Lustre: server umount lustre-MDT0000 complete [12536.656781] LustreError: 403469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872229 with bad export cookie 6915617171032001734 [12536.659812] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12536.664719] LustreError: 403469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12537.258754] Lustre: server umount lustre-MDT0001 complete [12547.062634] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12549.232994] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:10:41 (1787872241) [12556.677156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12557.332082] LustreError: 407910:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12557.349234] LustreError: 407910:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12557.449281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12562.694586] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12562.922471] LustreError: 407909:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12567.024711] LustreError: 407910:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12572.142684] LustreError: 407909:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12572.877473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12573.467490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12578.577470] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12586.968651] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12593.156572] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12601.665552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12602.234125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12609.417871] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12618.042176] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12620.630431] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12623.190835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12628.963397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12628.975059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12628.980334] LustreError: Skipped 2 previous similar messages [12628.985753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12629.018554] Lustre: Skipped 1 previous similar message [12634.783825] Lustre: server umount lustre-OST0000 complete [12639.206428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12639.226388] Lustre: Skipped 1 previous similar message [12645.284291] Lustre: server umount lustre-MDT0000 complete [12649.444381] LustreError: 407910:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12649.466708] LustreError: 407910:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12649.869993] LustreError: 407889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872342 with bad export cookie 6915617171032002616 [12649.874579] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12649.889238] LustreError: 407889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12650.218883] Lustre: server umount lustre-MDT0001 complete [12663.458734] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [12667.404304] Key type lgssc unregistered [12667.802863] LNet: 411646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12667.819457] LNetError: 411646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12667.837031] LNet: Removed LNI 192.168.202.131@tcp [12668.969750] Key type .llcrypt unregistered [12668.973336] Key type ._llcrypt unregistered [12697.563237] Key type ._llcrypt registered [12697.567972] Key type .llcrypt registered [12697.680261] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [12698.062308] Key type .llcrypt unregistered [12698.064750] Key type ._llcrypt unregistered [12719.224996] Key type ._llcrypt registered [12719.228567] Key type .llcrypt registered [12719.331760] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [12720.498791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12720.535847] alg: No test for adler32 (adler32-zlib) [12721.675277] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [12721.871485] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [12723.560573] Key type lgssc registered [12724.739930] Lustre: Echo OBD driver; http://www.lustre.org/ [12734.247412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12734.260747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12735.913843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12741.492660] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12745.194647] LustreError: 413866:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12745.229506] LustreError: 413866:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12750.317682] LustreError: 413867:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12750.974820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12751.319396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12756.459376] LustreError: 413866:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12756.476649] LustreError: 413866:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12758.857735] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12760.565967] LustreError: 413867:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12760.593415] LustreError: 413867:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12760.595089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12765.666473] LustreError: 413866:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12765.684996] LustreError: 413866:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12766.828019] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12774.340423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12774.643920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12779.866653] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12790.241623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12790.266102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12790.278622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12790.303139] Lustre: Skipped 1 previous similar message [12795.141178] Lustre: server umount lustre-OST0000 complete [12800.481541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12800.486196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12800.491337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12800.499537] Lustre: Skipped 2 previous similar messages [12805.330788] Lustre: server umount lustre-MDT0000 complete [12805.606653] LustreError: 413866:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12805.665020] LustreError: 413866:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12810.466735] LustreError: 413845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872503 with bad export cookie 17693950894730736567 [12810.467989] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12810.474296] LustreError: 413845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12811.575713] Lustre: server umount lustre-MDT0001 complete [12825.799786] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [12828.248960] Key type lgssc unregistered [12828.442566] LNet: 417018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12828.445430] LNetError: 417018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12828.454826] LNet: Removed LNI 192.168.202.131@tcp [12829.014273] Key type .llcrypt unregistered [12829.015558] Key type ._llcrypt unregistered [12859.367465] Key type ._llcrypt registered [12859.370374] Key type .llcrypt registered [12859.520879] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [12861.824202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12861.893373] alg: No test for adler32 (adler32-zlib) [12863.073324] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [12863.470317] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [12865.257424] Key type lgssc registered [12866.939828] Lustre: Echo OBD driver; http://www.lustre.org/ [12875.965614] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12875.996913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12891.680452] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12896.800336] LustreError: 418408:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [12923.552194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12929.836951] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12939.633593] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12948.528961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12949.173231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12949.261633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12952.922766] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12954.597957] LustreError: 418434:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12954.633196] LustreError: 418434:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12958.691643] LustreError: 418433:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12958.711092] LustreError: 418433:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12961.722292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12962.440489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12968.604509] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [12977.805472] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12983.571502] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12990.434445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12990.438259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12990.461333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12991.969971] Lustre: server umount lustre-OST0000 complete [12997.094326] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12997.102509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12997.130631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12998.634457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12998.638321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12998.678910] Lustre: Skipped 1 previous similar message [13003.751424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13003.758578] Lustre: Skipped 1 previous similar message [13008.866280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13008.874978] Lustre: Skipped 1 previous similar message [13009.889679] Lustre: 417747:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787872686/real 1787872686] req@00000000f9a55eb7 x1874720253312768/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787872702 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13009.890709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13009.920762] Lustre: 417747:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [13010.400811] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13010.588139] Lustre: server umount lustre-MDT0000 complete [13014.273705] LustreError: 418414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872707 with bad export cookie 5322104357159980174 [13014.285037] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13014.833127] Lustre: server umount lustre-MDT0001 complete [13031.115576] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [13035.631507] Key type lgssc unregistered [13036.024280] LNet: 421983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13036.034405] LNetError: 421983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13036.048158] LNet: Removed LNI 192.168.202.131@tcp [13037.030804] Key type .llcrypt unregistered [13037.032612] Key type ._llcrypt unregistered [13062.145076] Key type ._llcrypt registered [13062.150630] Key type .llcrypt registered [13062.309912] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [13063.302253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13063.411753] alg: No test for adler32 (adler32-zlib) [13064.565187] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [13064.884503] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [13066.579340] Key type lgssc registered [13067.704073] Lustre: Echo OBD driver; http://www.lustre.org/ [13076.140569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13076.159177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13077.677562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13083.064643] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13087.203729] LustreError: 423402:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13087.219585] LustreError: 423402:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13092.336677] LustreError: 423403:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13092.666946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13093.128405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13098.514895] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13106.075711] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13112.524229] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13119.970650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13120.313707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13122.374197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13127.046555] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13135.483474] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13143.449852] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [13144.034403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13144.038464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13144.044857] Lustre: Skipped 2 previous similar messages [13144.058559] Lustre: Skipped 2 previous similar messages [13147.151391] Lustre: server umount lustre-MDT0000 complete [13148.537410] LustreError: 425983:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13150.022654] LustreError: 423384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.31@tcp arrived at 1787872843 with bad export cookie 15733664515188430211 [13150.035050] LustreError: 423384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13152.226130] LustreError: 425245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872845 with bad export cookie 15733664515188429322 [13152.236265] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13152.244572] LustreError: 425245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13154.273582] LustreError: 425983:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13154.275576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13154.300414] LustreError: 425983:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [13154.314285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13154.349876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13157.282463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13158.666148] Lustre: server umount lustre-OST0000 complete [13162.404465] LustreError: 423403:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13162.420194] LustreError: 423403:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13165.205155] Lustre: server umount lustre-MDT0001 complete [13176.401861] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [13179.380785] Key type lgssc unregistered [13179.828338] LNet: 427091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13179.834084] LNetError: 427091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13180.906261] LNet: Removed LNI 192.168.202.131@tcp [13181.876397] Key type .llcrypt unregistered [13181.880781] Key type ._llcrypt unregistered [13208.737396] Key type ._llcrypt registered [13208.738777] Key type .llcrypt registered [13208.867957] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [13210.273829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13210.358129] alg: No test for adler32 (adler32-zlib) [13211.455725] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [13211.753317] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [13213.456188] Key type lgssc registered [13215.160234] Lustre: Echo OBD driver; http://www.lustre.org/ [13224.159482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13224.179206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13225.558254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13230.664552] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13235.683900] LustreError: 428509:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13235.715104] LustreError: 428509:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13240.501215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13240.801582] LustreError: 428510:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13241.146966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13247.006510] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13255.386922] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13262.271740] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13269.271318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13269.682739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13270.731575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13276.468889] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13285.145824] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13292.515907] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13292.517156] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13292.548689] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13292.563433] Lustre: Skipped 1 previous similar message [13296.406597] Lustre: server umount lustre-MDT0001 complete [13297.645702] LustreError: 428504:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13297.678746] LustreError: 428504:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13302.755482] LustreError: 430352:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13302.774093] LustreError: 430352:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13306.849390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13306.858833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13306.871458] Lustre: Skipped 1 previous similar message [13306.877284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13306.883386] Lustre: Skipped 1 previous similar message [13307.878437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13307.881721] LustreError: 428509:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13310.057410] Lustre: server umount lustre-OST0000 complete [13314.707157] Lustre: server umount lustre-MDT0000 complete [13330.601706] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [13334.065830] Key type lgssc unregistered [13334.511118] LNet: 432198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13334.516276] LNetError: 432198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13334.542883] LNet: Removed LNI 192.168.202.131@tcp [13335.515708] Key type .llcrypt unregistered [13335.517350] Key type ._llcrypt unregistered [13365.724724] Key type ._llcrypt registered [13365.727981] Key type .llcrypt registered [13365.870140] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [13381.531843] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [13382.825758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13382.849583] alg: No test for adler32 (adler32-zlib) [13384.154378] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [13384.393724] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [13386.049651] Key type lgssc registered [13387.290975] Lustre: Echo OBD driver; http://www.lustre.org/ [13428.897527] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [13446.052902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13446.084115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13447.479445] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13447.545416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13447.719713] Lustre: lustre-MDT0000: new disk, initializing [13447.863763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13447.885047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13453.291814] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13470.938268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13471.327865] Lustre: lustre-OST0000: new disk, initializing [13471.332132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13471.341090] Lustre: Skipped 1 previous similar message [13471.357628] Lustre: 437236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13471.491226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13479.321900] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13479.568601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13479.594216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13479.661769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13489.572536] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13508.332776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13508.574062] Lustre: 436216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13508.669137] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13508.787567] Lustre: lustre-MDT0001: new disk, initializing [13508.987504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13509.057837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13509.083779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13513.129847] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13524.264622] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13524.512577] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13529.616262] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13529.903369] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13538.686138] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.31@tcp (stopping) [13539.298678] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13539.311226] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13539.318783] Lustre: Skipped 1 previous similar message [13541.028388] Lustre: server umount lustre-MDT0001 complete [13549.567904] LustreError: 436225:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13549.595769] LustreError: 436225:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [13554.665172] LustreError: 436222:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13554.691225] LustreError: 436222:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13559.795624] LustreError: 438011:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13559.853255] LustreError: 438011:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13561.827232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13561.841417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13561.864931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13561.876500] Lustre: Skipped 1 previous similar message [13564.898204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13564.900204] LustreError: 436229:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13566.631835] Lustre: server umount lustre-OST0000 complete [13570.029714] LustreError: 436228:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13585.376181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13585.720290] Lustre: server umount lustre-MDT0000 complete [13601.640148] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [13605.115182] Key type lgssc unregistered [13605.571898] LNet: 440990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13605.577430] LNetError: 440990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13605.591862] LNet: Removed LNI 192.168.202.131@tcp [13606.556173] Key type .llcrypt unregistered [13606.557761] Key type ._llcrypt unregistered [13653.126826] Key type ._llcrypt registered [13653.129459] Key type .llcrypt registered [13653.281260] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [13653.631337] Key type .llcrypt unregistered [13653.634191] Key type ._llcrypt unregistered [13672.777987] Key type ._llcrypt registered [13672.780507] Key type .llcrypt registered [13672.944217] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [13674.490715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13674.511605] alg: No test for adler32 (adler32-zlib) [13675.625898] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [13675.915428] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [13677.664961] Key type lgssc registered [13678.959108] Lustre: Echo OBD driver; http://www.lustre.org/ [13710.392837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13710.420027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13711.825970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13711.863933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13711.963979] Lustre: lustre-MDT0000: new disk, initializing [13712.059570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13712.081963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13717.376406] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13728.721041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13728.827251] Lustre: 444377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13728.878300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13728.881428] Lustre: Skipped 1 previous similar message [13728.935241] Lustre: lustre-MDT0001: new disk, initializing [13729.018050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13729.060444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13729.076178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13733.961335] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13744.846295] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13751.165346] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13758.311313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13758.697729] Lustre: lustre-OST0000: new disk, initializing [13758.708616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13758.715134] Lustre: 446345:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13758.812467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13766.488571] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13766.765729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13766.785812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13767.003067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13778.942952] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13787.627395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13787.632341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13787.655634] Lustre: Skipped 2 previous similar messages [13787.663373] Lustre: Skipped 2 previous similar messages [13792.742158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13792.749621] Lustre: Skipped 2 previous similar messages [13793.358792] Lustre: server umount lustre-MDT0000 complete [13797.769086] LustreError: 444370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873490 with bad export cookie 12391619482049963386 [13797.776624] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13797.778418] LustreError: 444370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13797.865128] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13797.884926] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13798.269264] Lustre: server umount lustre-MDT0001 complete [13803.185954] Lustre: server umount lustre-OST0000 complete [13814.405714] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 19:31:46 (1787873506) [13822.239773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13822.585476] LustreError: 448373:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13822.606282] LustreError: 448373:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13822.691940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13828.019626] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13828.065904] LustreError: 448374:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13832.176969] LustreError: 448373:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13837.133381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13837.281804] LustreError: 448374:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13837.743392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13843.610463] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13852.041647] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13857.993780] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13877.510984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13877.951055] Lustre: Found index 0 for lustre-OST0000, updating log [13878.291456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13885.616682] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [13893.695982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13894.157738] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13902.827160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13902.834380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13902.849200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13909.988410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13909.997308] Lustre: Skipped 3 previous similar messages [13914.080160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13914.258723] Lustre: server umount lustre-OST0000 complete [13931.498798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13931.506200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13931.523459] Lustre: Skipped 2 previous similar messages [13931.535068] Lustre: Skipped 2 previous similar messages [13936.536638] Lustre: server umount lustre-MDT0000 complete [13936.612180] LustreError: 449318:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13936.633495] LustreError: 449318:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13938.086687] LNet: 1 local NIs in recovery (showing 1): 192.168.202.131@tcp [13940.684940] LustreError: 448355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873633 with bad export cookie 12391619482049964779 [13940.693702] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13940.719303] LustreError: 448355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13941.152760] Lustre: server umount lustre-MDT0001 complete [13951.204281] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 19:34:03 (1787873643) [14576.880450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14577.234474] LustreError: 454785:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14577.295887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14582.755356] LustreError: 454786:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14583.023673] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [14587.875903] LustreError: 454785:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14590.334653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14590.711311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14596.427856] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [14602.176662] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14607.347643] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14627.062067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14627.469405] Lustre: Found index 0 for lustre-OST0000, updating log [14627.882678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14629.345691] LNet: 1 local NIs in recovery (showing 1): 192.168.202.131@tcp [14634.536283] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [14642.117861] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14644.691167] Lustre: server umount lustre-OST0000 complete [14649.514655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14649.536445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14649.543964] Lustre: Skipped 1 previous similar message [14651.620138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14651.638968] Lustre: Skipped 3 previous similar messages [14654.691511] LustreError: 455984:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14654.705632] LustreError: 455984:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [14654.740187] Lustre: server umount lustre-MDT0000 complete [14658.764352] LustreError: 454765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874351 with bad export cookie 12391619482049965864 [14658.768678] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14658.785816] LustreError: 454765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14659.119744] Lustre: server umount lustre-MDT0001 complete [14689.762760] LNet: 1 local NIs in recovery (showing 1): 192.168.202.131@tcp [14690.899914] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [14701.478638] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [14736.263803] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 19:47:08 (1787874428) [14753.056279] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [14756.184304] Key type lgssc unregistered [14756.544537] LNet: 461398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14756.551833] LNetError: 461398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14882.534787] LNet: Removed LNI 192.168.202.131@tcp [14883.540228] Key type .llcrypt unregistered [14883.541834] Key type ._llcrypt unregistered [14900.502977] Key type ._llcrypt registered [14900.504864] Key type .llcrypt registered [14900.598685] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [14901.696412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14901.734180] alg: No test for adler32 (adler32-zlib) [14903.055836] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [14903.380384] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [14905.048918] Key type lgssc registered [14906.306969] Lustre: Echo OBD driver; http://www.lustre.org/ [14910.671233] LNet: Added LNI 192.168.202.131@tcp100 [8/256/0/180] [14911.722314] LNet: Added LNI 192.168.202.131@tcp101 [8/256/0/180] [14912.805431] LNet: Added LNI 192.168.202.131@tcp102 [8/256/0/180] [14913.728440] LNet: Added LNI 192.168.202.131@tcp103 [8/256/0/180] [14914.580387] LNet: Added LNI 192.168.202.131@tcp104 [8/256/0/180] [14915.657485] LNet: Added LNI 192.168.202.131@tcp105 [8/256/0/180] [14916.653165] LNet: Added LNI 192.168.202.131@tcp106 [8/256/0/180] [14917.581951] LNet: Added LNI 192.168.202.131@tcp107 [8/256/0/180] [14918.543672] LNet: Added LNI 192.168.202.131@tcp108 [8/256/0/180] [14919.515199] LNet: Added LNI 192.168.202.131@tcp109 [8/256/0/180] [14920.532863] LNet: Added LNI 192.168.202.131@tcp110 [8/256/0/180] [14921.522859] LNet: Added LNI 192.168.202.131@tcp111 [8/256/0/180] [14922.434752] LNet: Added LNI 192.168.202.131@tcp112 [8/256/0/180] [14923.144690] LNet: Added LNI 192.168.202.131@tcp113 [8/256/0/180] [14924.134767] LNet: Added LNI 192.168.202.131@tcp114 [8/256/0/180] [14924.939790] LNet: Added LNI 192.168.202.131@tcp115 [8/256/0/180] [14925.843385] LNet: Added LNI 192.168.202.131@tcp116 [8/256/0/180] [14926.717321] LNet: Added LNI 192.168.202.131@tcp117 [8/256/0/180] [14927.835283] LNet: Added LNI 192.168.202.131@tcp118 [8/256/0/180] [14928.874312] LNet: Added LNI 192.168.202.131@tcp119 [8/256/0/180] [14929.855221] LNet: Added LNI 192.168.202.131@tcp120 [8/256/0/180] [14930.870569] LNet: Added LNI 192.168.202.131@tcp121 [8/256/0/180] [14931.946166] LNet: Added LNI 192.168.202.131@tcp122 [8/256/0/180] [14932.918528] LNet: Added LNI 192.168.202.131@tcp123 [8/256/0/180] [14933.873787] LNet: Added LNI 192.168.202.131@tcp124 [8/256/0/180] [14934.751410] LNet: Added LNI 192.168.202.131@tcp125 [8/256/0/180] [14935.752242] LNet: Added LNI 192.168.202.131@tcp126 [8/256/0/180] [14936.858286] LNet: Added LNI 192.168.202.131@tcp127 [8/256/0/180] [14937.774880] LNet: Added LNI 192.168.202.131@tcp128 [8/256/0/180] [14938.569329] LNet: Added LNI 192.168.202.131@tcp129 [8/256/0/180] [14939.599819] LNet: Added LNI 192.168.202.131@tcp130 [8/256/0/180] [14940.399446] LNet: Added LNI 192.168.202.131@tcp131 [8/256/0/180] [14941.305253] LNet: Added LNI 192.168.202.131@tcp132 [8/256/0/180] [14942.021421] LNet: Added LNI 192.168.202.131@tcp133 [8/256/0/180] [14943.130204] LNet: Added LNI 192.168.202.131@tcp134 [8/256/0/180] [14944.153886] LNet: Added LNI 192.168.202.131@tcp135 [8/256/0/180] [14951.234407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14951.301734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14952.444803] Lustre: 464433:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.131@tcp: can't add NID 192.168.202.131@tcp130: rc = -75 [14952.555474] LNetError: 461893:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.131 [14952.572602] LNetError: Protocol error connecting to 192.168.202.131@tcp120 at host 192.168.202.131:988: is it running a compatible version of Lustre? [14952.594301] LNetError: Skipped 4 previous similar messages [14952.638387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14952.658739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14952.717581] Lustre: lustre-MDT0000: new disk, initializing [14952.802491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14952.826038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14953.568329] LNet: 6 local NIs in recovery (showing 6): 192.168.202.131@tcp124, 192.168.202.131@tcp125, 192.168.202.131@tcp126, 192.168.202.131@tcp129, 192.168.202.131@tcp127, 192.168.202.131@tcp120 [14953.572821] LNetError: 464454:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.131 [14953.610973] LNetError: 464454:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [14953.639282] LNetError: Protocol error connecting to 192.168.202.131@tcp100 at host 192.168.202.131:988: is it running a compatible version of Lustre? [14953.658631] LNetError: Skipped 1 previous similar message [14957.733160] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [14969.644039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14969.825624] Lustre: 464450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14969.937060] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14969.946329] Lustre: Skipped 1 previous similar message [14970.089913] Lustre: lustre-MDT0001: new disk, initializing [14970.155954] Lustre: 465250:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.131@tcp: can't add NID 192.168.202.131@tcp130: rc = -75 [14970.169534] Lustre: 465250:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14970.316689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14970.395178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14970.411573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14975.272412] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [14978.675970] Lustre: 465619:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.131@tcp: can't add NID 192.168.202.131@tcp130: rc = -75 [14978.687607] Lustre: 465619:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14982.248134] LNetError: 464464:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.131 [14982.261332] LNetError: 464464:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14982.282627] LNetError: Protocol error connecting to 192.168.202.131@tcp100 at host 192.168.202.131:988: is it running a compatible version of Lustre? [14985.721565] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14989.415480] LNetError: 461894:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.131 [14989.443920] LNetError: Protocol error connecting to 192.168.202.131@tcp125 at host 192.168.202.131:988: is it running a compatible version of Lustre? [14991.767747] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14993.506189] LNetError: 461892:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.131 [14993.529718] LNetError: 461892:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14993.552148] LNetError: Protocol error connecting to 192.168.202.131@tcp118 at host 192.168.202.131:988: is it running a compatible version of Lustre? [14993.577367] LNetError: Skipped 2 previous similar messages [14999.131887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14999.725628] Lustre: lustre-OST0000: new disk, initializing [14999.733812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14999.747261] Lustre: 466427:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14999.871244] Lustre: 466432:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.131@tcp: can't add NID 192.168.202.131@tcp130: rc = -75 [14999.884902] Lustre: 466432:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14999.935907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15002.729775] LNetError: 464453:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.131 [15002.746654] LNetError: 464453:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15002.752465] LNetError: Protocol error connecting to 192.168.202.131@tcp125 at host 192.168.202.131:988: is it running a compatible version of Lustre? [15002.763754] LNetError: Skipped 1 previous similar message [15005.277066] Lustre: 466696:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.131@tcp: can't add NID 192.168.202.131@tcp130: rc = -75 [15005.291552] Lustre: 466696:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15005.369548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15005.401709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15005.577992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15006.858550] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [15017.315097] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15019.113734] LNetError: 461892:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.131 [15019.124208] LNetError: 461892:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [15019.138619] LNetError: Protocol error connecting to 192.168.202.131@tcp118 at host 192.168.202.131:988: is it running a compatible version of Lustre? [15019.150459] LNetError: Skipped 7 previous similar messages [15026.152691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15026.159690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15026.175985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15026.181325] Lustre: Skipped 1 previous similar message [15026.196312] Lustre: Skipped 1 previous similar message [15031.174636] Lustre: server umount lustre-OST0000 complete [15036.396208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15036.406835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15036.419614] Lustre: Skipped 1 previous similar message [15036.437236] Lustre: Skipped 1 previous similar message [15041.506315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15041.516157] Lustre: Skipped 1 previous similar message [15042.068677] Lustre: server umount lustre-MDT0000 complete [15045.224812] LustreError: 466426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874738 with bad export cookie 6327363879192835522 [15045.229428] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15045.231535] LustreError: 466426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15045.540751] Lustre: server umount lustre-MDT0001 complete [15053.933376] LNetError: 464464:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.131 [15053.952148] LNetError: 464464:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [15053.966500] LNetError: Protocol error connecting to 192.168.202.131@tcp125 at host 192.168.202.131:988: is it running a compatible version of Lustre? [15053.999023] LNetError: Skipped 5 previous similar messages [15057.702354] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [15061.812187] Key type lgssc unregistered [15062.345911] LNet: 468344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15062.369583] LNetError: 468344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15062.381904] LNet: Removed LNI 192.168.202.131@tcp [15063.392209] LNet: Removed LNI 192.168.202.131@tcp100 [15063.399699] LNetError: 468344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15063.441408] LNetError: 468344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15063.474245] LNet: Removed LNI 192.168.202.131@tcp101 [15064.480497] LNet: Removed LNI 192.168.202.131@tcp102 [15064.510266] LNetError: 468344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15064.521620] LNetError: 468344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15064.525479] LNet: Removed LNI 192.168.202.131@tcp103 [15065.568222] LNet: Removed LNI 192.168.202.131@tcp104 [15066.592415] LNet: Removed LNI 192.168.202.131@tcp105 [15066.594080] LNetError: 468344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15066.603903] LNetError: 468344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15066.624179] LNet: Removed LNI 192.168.202.131@tcp106 [15067.680894] LNet: Removed LNI 192.168.202.131@tcp107 [15068.705073] LNet: Removed LNI 192.168.202.131@tcp108 [15069.734489] LNet: Removed LNI 192.168.202.131@tcp109 [15070.753794] LNet: Removed LNI 192.168.202.131@tcp110 [15070.756265] LNetError: 468344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15070.779080] LNetError: 468344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15070.792957] LNet: Removed LNI 192.168.202.131@tcp111 [15071.840273] LNet: Removed LNI 192.168.202.131@tcp112 [15072.864520] LNet: Removed LNI 192.168.202.131@tcp113 [15073.888145] LNet: Removed LNI 192.168.202.131@tcp114 [15074.913208] LNet: Removed LNI 192.168.202.131@tcp115 [15075.936575] LNet: Removed LNI 192.168.202.131@tcp116 [15076.960318] LNet: Removed LNI 192.168.202.131@tcp117 [15077.984661] LNet: Removed LNI 192.168.202.131@tcp118 [15079.008342] LNet: Removed LNI 192.168.202.131@tcp119 [15079.013230] LNetError: 468344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15079.021761] LNetError: 468344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15079.047893] LNet: Removed LNI 192.168.202.131@tcp120 [15080.097023] LNet: Removed LNI 192.168.202.131@tcp121 [15081.123350] LNet: Removed LNI 192.168.202.131@tcp122 [15082.144496] LNet: Removed LNI 192.168.202.131@tcp123 [15083.168661] LNet: Removed LNI 192.168.202.131@tcp124 [15084.201151] LNet: Removed LNI 192.168.202.131@tcp125 [15085.219138] LNet: Removed LNI 192.168.202.131@tcp126 [15086.240193] LNet: Removed LNI 192.168.202.131@tcp127 [15087.264552] LNet: Removed LNI 192.168.202.131@tcp128 [15088.352670] LNet: Removed LNI 192.168.202.131@tcp129 [15088.361058] LNet: Removed LNI 192.168.202.131@tcp130 [15088.376053] LNet: Removed LNI 192.168.202.131@tcp131 [15088.394553] LNet: Removed LNI 192.168.202.131@tcp132 [15088.400729] LNet: Removed LNI 192.168.202.131@tcp133 [15088.409111] LNet: Removed LNI 192.168.202.131@tcp134 [15088.438962] LNet: Removed LNI 192.168.202.131@tcp135 [15089.437373] Key type .llcrypt unregistered [15089.440413] Key type ._llcrypt unregistered [15114.405475] Key type ._llcrypt registered [15114.408587] Key type .llcrypt registered [15114.552900] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [15130.308466] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [15131.413211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15131.568972] alg: No test for adler32 (adler32-zlib) [15132.741776] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [15133.121477] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [15134.824320] Key type lgssc registered [15137.160206] Lustre: Echo OBD driver; http://www.lustre.org/ [15175.263622] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 19:54:27 (1787874867) [15194.221042] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [15196.342178] Key type lgssc unregistered [15196.585772] LNet: 472436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15196.591976] LNetError: 472436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15196.603885] LNet: Removed LNI 192.168.202.131@tcp [15197.297794] Key type .llcrypt unregistered [15197.300428] Key type ._llcrypt unregistered [15213.621824] Key type ._llcrypt registered [15213.624454] Key type .llcrypt registered [15213.768963] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [15215.886817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15215.911199] alg: No test for adler32 (adler32-zlib) [15217.055508] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [15217.322939] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [15219.072247] Key type lgssc registered [15220.854132] Lustre: Echo OBD driver; http://www.lustre.org/ [15231.613689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15231.656285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15233.030417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15233.060464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15233.139378] Lustre: lustre-MDT0000: new disk, initializing [15233.266280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15233.305620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15238.590616] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [15249.520192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15249.709554] Lustre: 473651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15249.897241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15249.916239] Lustre: Skipped 1 previous similar message [15250.079413] Lustre: lustre-MDT0001: new disk, initializing [15250.217506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15250.266107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15250.287249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15255.075290] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [15265.949178] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15272.307129] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15279.476703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15279.898202] Lustre: lustre-OST0000: new disk, initializing [15279.901983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15279.909254] Lustre: 475621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15280.052384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15285.393132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15285.406230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15285.522485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15287.347173] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [15298.177283] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15302.484617] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing lnet_if_list [15308.478385] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15308.513342] LNet: Added LNI 192.168.202.131@tcp10 [8/256/0/180] [15315.435438] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15315.471372] LNet: Added LNI 192.168.202.131@tcp11 [8/256/0/180] [15320.454453] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15320.486720] LNet: Added LNI 192.168.202.131@tcp12 [8/256/0/180] [15324.033411] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15324.056971] LNet: Added LNI 192.168.202.131@tcp13 [8/256/0/180] [15328.115664] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15328.138809] LNet: Added LNI 192.168.202.131@tcp14 [8/256/0/180] [15333.204159] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15333.228430] LNet: Added LNI 192.168.202.131@tcp15 [8/256/0/180] [15337.988541] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15338.028816] LNet: Added LNI 192.168.202.131@tcp16 [8/256/0/180] [15343.452641] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15343.485568] LNet: Added LNI 192.168.202.131@tcp17 [8/256/0/180] [15348.199777] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15348.224666] LNet: Added LNI 192.168.202.131@tcp18 [8/256/0/180] [15353.236714] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15353.268884] LNet: Added LNI 192.168.202.131@tcp19 [8/256/0/180] [15358.514363] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15358.550315] LNet: Added LNI 192.168.202.131@tcp20 [8/256/0/180] [15364.837511] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15364.867135] LNet: Added LNI 192.168.202.131@tcp21 [8/256/0/180] [15370.664099] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15370.727740] LNet: Added LNI 192.168.202.131@tcp22 [8/256/0/180] [15376.996089] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15377.051167] LNet: Added LNI 192.168.202.131@tcp23 [8/256/0/180] [15383.105598] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15383.131166] LNet: Added LNI 192.168.202.131@tcp24 [8/256/0/180] [15388.792691] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15388.808438] LNet: Added LNI 192.168.202.131@tcp25 [8/256/0/180] [15395.972758] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15396.006624] LNet: Added LNI 192.168.202.131@tcp26 [8/256/0/180] [15402.424972] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15402.456292] LNet: Added LNI 192.168.202.131@tcp27 [8/256/0/180] [15410.396735] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15410.437435] LNet: Added LNI 192.168.202.131@tcp28 [8/256/0/180] [15418.318326] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15418.359500] LNet: Added LNI 192.168.202.131@tcp29 [8/256/0/180] [15426.362945] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15426.427713] LNet: Added LNI 192.168.202.131@tcp30 [8/256/0/180] [15433.406997] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15433.422304] LNet: Added LNI 192.168.202.131@tcp31 [8/256/0/180] [15440.588070] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15440.634293] LNet: Added LNI 192.168.202.131@tcp32 [8/256/0/180] [15449.764873] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15449.833660] LNet: Added LNI 192.168.202.131@tcp33 [8/256/0/180] [15457.964278] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15457.991435] LNet: Added LNI 192.168.202.131@tcp34 [8/256/0/180] [15466.425768] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15466.463051] LNet: Added LNI 192.168.202.131@tcp35 [8/256/0/180] [15474.641748] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15474.693956] LNet: Added LNI 192.168.202.131@tcp36 [8/256/0/180] [15482.778785] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15482.800489] LNet: Added LNI 192.168.202.131@tcp37 [8/256/0/180] [15490.884771] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15490.914530] LNet: Added LNI 192.168.202.131@tcp38 [8/256/0/180] [15498.791050] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15498.839024] LNet: Added LNI 192.168.202.131@tcp39 [8/256/0/180] [15506.989670] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15507.002669] LNet: Added LNI 192.168.202.131@tcp40 [8/256/0/180] [15515.291623] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15515.321718] LNet: Added LNI 192.168.202.131@tcp41 [8/256/0/180] [15522.534187] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15522.575588] LNet: Added LNI 192.168.202.131@tcp42 [8/256/0/180] [15532.939099] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15533.001852] LNet: Added LNI 192.168.202.131@tcp43 [8/256/0/180] [15541.458565] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15541.484870] LNet: Added LNI 192.168.202.131@tcp44 [8/256/0/180] [15547.179810] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15547.237200] LNet: Added LNI 192.168.202.131@tcp45 [8/256/0/180] [15553.808491] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15553.841291] LNet: Added LNI 192.168.202.131@tcp46 [8/256/0/180] [15562.262471] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15562.281463] LNet: Added LNI 192.168.202.131@tcp47 [8/256/0/180] [15571.643151] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15571.655175] LNet: Added LNI 192.168.202.131@tcp48 [8/256/0/180] [15584.905849] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15584.985433] LNet: Added LNI 192.168.202.131@tcp49 [8/256/0/180] [15600.520028] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15600.582547] LNet: Added LNI 192.168.202.131@tcp50 [8/256/0/180] [15639.031858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15639.037967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15639.059448] Lustre: Skipped 1 previous similar message [15642.241118] Lustre: server umount lustre-MDT0000 complete [15644.132932] LustreError: 476102:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15644.171287] LustreError: 476102:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [15646.617538] LustreError: 473644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875339 with bad export cookie 9256371545806532469 [15646.634127] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15647.358152] Lustre: server umount lustre-MDT0001 complete [15665.575528] Lustre: 472924:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787875342/real 1787875342] req@000000002699b858 x1874722721817344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787875358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15665.600994] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15665.610193] Lustre: Skipped 1 previous similar message [15667.686831] Lustre: 472923:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787875344/real 1787875344] req@000000005d310ad6 x1874722721817472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787875360 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15667.787345] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [15670.752607] Lustre: 472922:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787875347/real 1787875347] req@000000003f83f42f x1874722721817600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787875363 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15677.454045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15677.920282] Lustre: 472925:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787875354/real 1787875354] req@000000002b0007a6 x1874722721817984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787875370 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15677.955787] Lustre: 472925:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15691.513758] LustreError: 475614:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15691.584411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15694.626437] LustreError: 475615:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15694.629074] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15696.786758] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [15700.963427] LustreError: 475614:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15700.987104] LustreError: 475614:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15706.011179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15706.085349] LustreError: 475615:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15706.377785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15710.533841] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15711.884236] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [15720.299524] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15725.595957] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15742.945428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15742.955651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15742.970219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15742.972948] Lustre: Skipped 2 previous similar messages [15746.532595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15746.534362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15751.672165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15751.675976] Lustre: Skipped 2 previous similar messages [15756.771077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15757.280219] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15757.607435] Lustre: server umount lustre-OST0000 complete [15761.890157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15761.915921] Lustre: Skipped 1 previous similar message [15767.010717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15767.021182] Lustre: Skipped 4 previous similar messages [15768.036770] Lustre: server umount lustre-MDT0000 complete [15771.732422] LustreError: 475620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875464 with bad export cookie 9256371545806535115 [15771.735495] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15771.752796] LustreError: 475620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [15772.166504] Lustre: server umount lustre-MDT0001 complete [15785.676418] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [15788.149455] Key type lgssc unregistered [15788.503576] LNet: 502686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15788.510416] LNetError: 502686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15788.520993] LNet: Removed LNI 192.168.202.131@tcp [15788.523745] LNet: Removed LNI 192.168.202.131@tcp10 [15788.526347] LNet: Removed LNI 192.168.202.131@tcp11 [15788.528633] LNet: Removed LNI 192.168.202.131@tcp12 [15788.530859] LNet: Removed LNI 192.168.202.131@tcp13 [15788.532380] LNet: Removed LNI 192.168.202.131@tcp14 [15788.534643] LNet: Removed LNI 192.168.202.131@tcp15 [15788.537443] LNet: Removed LNI 192.168.202.131@tcp16 [15788.539431] LNet: Removed LNI 192.168.202.131@tcp17 [15788.542346] LNet: Removed LNI 192.168.202.131@tcp18 [15788.544448] LNet: Removed LNI 192.168.202.131@tcp19 [15788.546744] LNet: Removed LNI 192.168.202.131@tcp20 [15788.549453] LNet: Removed LNI 192.168.202.131@tcp21 [15788.551895] LNet: Removed LNI 192.168.202.131@tcp22 [15788.556307] LNet: Removed LNI 192.168.202.131@tcp23 [15788.558463] LNet: Removed LNI 192.168.202.131@tcp24 [15788.560555] LNet: Removed LNI 192.168.202.131@tcp25 [15788.562557] LNet: Removed LNI 192.168.202.131@tcp26 [15788.564697] LNet: Removed LNI 192.168.202.131@tcp27 [15788.567601] LNet: Removed LNI 192.168.202.131@tcp28 [15788.569683] LNet: Removed LNI 192.168.202.131@tcp29 [15788.572207] LNet: Removed LNI 192.168.202.131@tcp30 [15788.574399] LNet: Removed LNI 192.168.202.131@tcp31 [15788.576630] LNet: Removed LNI 192.168.202.131@tcp32 [15788.578693] LNet: Removed LNI 192.168.202.131@tcp33 [15788.580728] LNet: Removed LNI 192.168.202.131@tcp34 [15788.583027] LNet: Removed LNI 192.168.202.131@tcp35 [15788.586715] LNet: Removed LNI 192.168.202.131@tcp36 [15788.590438] LNet: Removed LNI 192.168.202.131@tcp37 [15788.597938] LNet: Removed LNI 192.168.202.131@tcp38 [15788.603506] LNet: Removed LNI 192.168.202.131@tcp39 [15788.607867] LNet: Removed LNI 192.168.202.131@tcp40 [15788.617430] LNet: Removed LNI 192.168.202.131@tcp41 [15788.620150] LNet: Removed LNI 192.168.202.131@tcp42 [15788.623263] LNet: Removed LNI 192.168.202.131@tcp43 [15788.625420] LNet: Removed LNI 192.168.202.131@tcp44 [15788.627670] LNet: Removed LNI 192.168.202.131@tcp45 [15788.630264] LNet: Removed LNI 192.168.202.131@tcp46 [15788.632506] LNet: Removed LNI 192.168.202.131@tcp47 [15788.634770] LNet: Removed LNI 192.168.202.131@tcp48 [15788.637836] LNet: Removed LNI 192.168.202.131@tcp49 [15788.648429] LNet: Removed LNI 192.168.202.131@tcp50 [15789.288863] Key type .llcrypt unregistered [15789.290540] Key type ._llcrypt unregistered [15809.040436] Key type ._llcrypt registered [15809.049171] Key type .llcrypt registered [15809.149744] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [15824.115329] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [15825.265657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15825.399171] alg: No test for adler32 (adler32-zlib) [15826.595940] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [15826.939146] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [15828.620228] Key type lgssc registered [15829.639160] Lustre: Echo OBD driver; http://www.lustre.org/ [15864.835675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15864.864935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15866.151673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15866.221269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15866.320874] Lustre: lustre-MDT0000: new disk, initializing [15866.416761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15866.432361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15871.257371] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [15881.818255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15881.917625] Lustre: 505881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15881.965437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15881.971348] Lustre: Skipped 1 previous similar message [15882.066954] Lustre: lustre-MDT0001: new disk, initializing [15882.141680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15882.179213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15882.203250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15886.119843] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [15895.711403] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15902.157085] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15910.085693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15910.295887] Lustre: *** cfs_fail_loc=90f, val=0*** [15910.408806] Lustre: lustre-OST0000: new disk, initializing [15910.413221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15910.421543] Lustre: 507899:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15910.576977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15917.466413] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [15918.684737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15918.698189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15918.841298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15928.161558] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15939.555178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15939.560775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15939.563534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15939.576567] Lustre: Skipped 1 previous similar message [15939.604155] Lustre: Skipped 1 previous similar message [15941.591705] Lustre: server umount lustre-OST0000 complete [15948.770747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15948.776388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15948.786797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15949.794341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15949.794847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15949.816528] Lustre: Skipped 1 previous similar message [15952.073378] Lustre: server umount lustre-MDT0000 complete [15956.300981] LustreError: 505874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875649 with bad export cookie 16303163580913446990 [15956.308031] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15956.328418] LustreError: 505874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15956.789394] Lustre: server umount lustre-MDT0001 complete [15967.710076] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [15970.573377] Key type lgssc unregistered [15970.877530] LNet: 509808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15970.881425] LNetError: 509808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15970.897974] LNet: Removed LNI 192.168.202.131@tcp [15971.980214] Key type .llcrypt unregistered [15971.984831] Key type ._llcrypt unregistered [15987.889763] Key type ._llcrypt registered [15987.892333] Key type .llcrypt registered [15988.016976] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [15989.781483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15989.796470] alg: No test for adler32 (adler32-zlib) [15990.840702] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [15991.065051] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [15992.722966] Key type lgssc registered [15993.509671] Lustre: Echo OBD driver; http://www.lustre.org/ [16000.046250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16000.079379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16001.454767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16004.774325] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [16010.950035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16011.289380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16014.970403] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [16020.746322] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16025.357507] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16031.398975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16031.801987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16038.973444] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [16045.459831] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16060.342768] loop0: detected capacity change from 0 to 204800000 [16060.554272] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16060.567504] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16060.584605] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16066.457783] loop0: detected capacity change from 0 to 204800000 [16066.719864] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16066.738628] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16066.762918] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16072.081750] loop0: detected capacity change from 0 to 204800000 [16077.422611] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16077.582604] Lustre: fs15246-OST0000: new disk, initializing [16077.590112] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16077.595692] Lustre: 514655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16077.680247] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16083.414477] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [16092.894295] loop1: detected capacity change from 0 to 204800000 [16098.016848] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16098.118695] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16098.174880] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16098.228925] Lustre: fs15246-MDT0000: new disk, initializing [16098.369887] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16098.396835] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16098.521871] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16099.554051] Lustre: 515776:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000038d53a20 x1874723533269760/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787875808 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16103.504887] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [16108.708380] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16108.718080] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16108.740632] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16126.976481] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16126.996481] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16129.552275] Lustre: server umount fs15246-MDT0000 complete [16132.068972] LustreError: 513401:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16132.088615] LustreError: 513401:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [16137.203124] LustreError: 511055:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16138.994498] Lustre: server umount fs15246-OST0000 complete [16149.475257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16149.485136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16149.488171] LustreError: Skipped 1 previous similar message [16149.488954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16149.496310] Lustre: Skipped 1 previous similar message [16149.519400] Lustre: Skipped 1 previous similar message [16152.549157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16152.557419] Lustre: Skipped 1 previous similar message [16155.734351] Lustre: server umount lustre-OST0000 complete [16159.718876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16159.728060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16159.744976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16162.790530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16165.139306] Lustre: server umount lustre-MDT0000 complete [16167.793561] LustreError: 511039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875860 with bad export cookie 13939124668369988702 [16167.799807] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16167.805236] LustreError: 511039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16168.279799] Lustre: server umount lustre-MDT0001 complete [16176.984456] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [16179.481607] Key type lgssc unregistered [16179.729314] LNet: 518873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16179.735534] LNetError: 518873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16179.752657] LNet: Removed LNI 192.168.202.131@tcp [16180.541590] Key type .llcrypt unregistered [16180.542904] Key type ._llcrypt unregistered [16199.112791] Key type ._llcrypt registered [16199.115585] Key type .llcrypt registered [16199.209381] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [16200.363636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16200.475562] alg: No test for adler32 (adler32-zlib) [16201.520631] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [16201.730622] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [16203.408162] Key type lgssc registered [16204.478706] Lustre: Echo OBD driver; http://www.lustre.org/ [16247.500934] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [16256.028724] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [16298.235524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16298.276169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16299.674261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16299.745491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16299.897687] Lustre: lustre-MDT0000: new disk, initializing [16300.017618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16300.037722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16304.996045] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [16315.273348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16315.503399] Lustre: 523339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16315.577358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16315.579983] Lustre: Skipped 1 previous similar message [16315.637202] Lustre: lustre-MDT0001: new disk, initializing [16315.752368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16315.805355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16315.816202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16319.664944] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [16331.553938] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16338.638102] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16347.095357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16347.517971] Lustre: lustre-OST0000: new disk, initializing [16347.539135] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16347.565940] Lustre: 525309:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16347.793876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16354.435712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16354.461520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16354.633259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16356.063610] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [16364.811597] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16370.056832] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16370.284165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16376.156712] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16376.372585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16380.393958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16380.407355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16380.425269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16380.433747] Lustre: Skipped 1 previous similar message [16385.509799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16385.521023] Lustre: Skipped 1 previous similar message [16390.626472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16390.648069] Lustre: Skipped 1 previous similar message [16393.185173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16393.323411] Lustre: server umount lustre-OST0000 complete [16400.868047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16400.872217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16400.887595] Lustre: Skipped 2 previous similar messages [16400.901822] Lustre: Skipped 1 previous similar message [16405.987120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16406.006042] Lustre: Skipped 1 previous similar message [16408.143554] Lustre: server umount lustre-MDT0000 complete [16412.853434] LustreError: 526649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876105 with bad export cookie 6372672206590944797 [16412.866298] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16413.507258] Lustre: server umount lustre-MDT0001 complete [16437.734466] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [16449.461208] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [16482.458707] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16160 sec ========= 20:16:13 (1787876173) [16484.134204] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:16:16 (1787876176) === [16487.769466] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:16:19 (1787876179) === [16527.299768] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [16530.340885] Key type lgssc unregistered [16530.672528] LNet: 531974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16530.680552] LNetError: 531974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16530.701251] LNet: Removed LNI 192.168.202.131@tcp [16531.541861] Key type .llcrypt unregistered [16531.547304] Key type ._llcrypt unregistered