[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495954990 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002378] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009133] LSM: Security Framework initializing [ 0.010048] Yama: becoming mindful. [ 0.011031] SELinux: Initializing. [ 0.012056] *** VALIDATE selinux *** [ 0.020131] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024596] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027112] *** VALIDATE tmpfs *** [ 0.029228] *** VALIDATE proc *** [ 0.030237] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.032257] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035039] Spectre V2 : User space: Vulnerable [ 0.036009] Speculative Store Bypass: Vulnerable [ 0.038622] debug: unmapping init [mem 0xffffffff9b059000-0xffffffff9b060fff] [ 0.040171] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041701] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042022] ... version: 2 [ 0.043011] ... bit width: 48 [ 0.044008] ... generic registers: 4 [ 0.045009] ... value mask: 0000ffffffffffff [ 0.046011] ... max period: 00007fffffffffff [ 0.047010] ... fixed-purpose events: 3 [ 0.048009] ... event mask: 000000070000000f [ 0.049288] rcu: Hierarchical SRCU implementation. [ 0.051419] smp: Bringing up secondary CPUs ... [ 0.052593] x86: Booting SMP configuration: [ 0.053023] .... node #0, CPUs: #1 #2 #3 [ 0.058018] smp: Brought up 1 node, 4 CPUs [ 0.060010] smpboot: Max logical packages: 1 [ 0.061016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135529] node 0 deferred pages initialised in 70ms [ 0.138043] devtmpfs: initialized [ 0.139288] x86/mm: Memory block size: 128MB [ 0.141950] gcov: version magic: 0x41383552 [ 0.143155] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144094] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145247] pinctrl core: initialized pinctrl subsystem [ 0.146184] [ 0.146928] ************************************************************* [ 0.147012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148011] ** ** [ 0.149015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.150009] ** ** [ 0.151014] ** This means that this kernel is built to expose internal ** [ 0.152011] ** IOMMU data structures, which may compromise security on ** [ 0.153014] ** your system. ** [ 0.154013] ** ** [ 0.155017] ** If you see this message and you are not debugging the ** [ 0.156017] ** kernel, report this immediately to your vendor! ** [ 0.157013] ** ** [ 0.158012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159012] ************************************************************* [ 0.160746] NET: Registered protocol family 16 [ 0.161519] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.162072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.163057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.164708] cpuidle: using governor menu [ 0.166869] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.168572] PCI: Using configuration type 1 for base access [ 0.170230] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.181139] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.186030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.190080] cryptd: max_cpu_qlen set to 1000 [ 0.193248] ACPI: Added _OSI(Module Device) [ 0.194000] ACPI: Added _OSI(Processor Device) [ 0.195011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.198256] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.204553] ACPI: Interpreter enabled [ 0.206053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.207011] ACPI: Using IOAPIC for interrupt routing [ 0.209122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.212385] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.224289] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.227044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.230018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.233161] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.238975] acpiphp: Slot [2] registered [ 0.240173] acpiphp: Slot [5] registered [ 0.241092] acpiphp: Slot [6] registered [ 0.242097] acpiphp: Slot [7] registered [ 0.244110] acpiphp: Slot [8] registered [ 0.245088] acpiphp: Slot [9] registered [ 0.246098] acpiphp: Slot [10] registered [ 0.248086] acpiphp: Slot [3] registered [ 0.249083] acpiphp: Slot [4] registered [ 0.251087] acpiphp: Slot [11] registered [ 0.252116] acpiphp: Slot [12] registered [ 0.253130] acpiphp: Slot [13] registered [ 0.254086] acpiphp: Slot [14] registered [ 0.256081] acpiphp: Slot [15] registered [ 0.257087] acpiphp: Slot [16] registered [ 0.259065] acpiphp: Slot [17] registered [ 0.260102] acpiphp: Slot [18] registered [ 0.262088] acpiphp: Slot [19] registered [ 0.263174] acpiphp: Slot [20] registered [ 0.265085] acpiphp: Slot [21] registered [ 0.267096] acpiphp: Slot [22] registered [ 0.268157] acpiphp: Slot [23] registered [ 0.270090] acpiphp: Slot [24] registered [ 0.271087] acpiphp: Slot [25] registered [ 0.273183] acpiphp: Slot [26] registered [ 0.274094] acpiphp: Slot [27] registered [ 0.276087] acpiphp: Slot [28] registered [ 0.278112] acpiphp: Slot [29] registered [ 0.279088] acpiphp: Slot [30] registered [ 0.281119] acpiphp: Slot [31] registered [ 0.282070] PCI host bridge to bus 0000:00 [ 0.284016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.286020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.292019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.294049] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.297026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.299284] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.302023] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305326] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.315719] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.321060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.325020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.327013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336567] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.338882] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.342041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344758] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.350016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.364017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.369013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.375012] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.388014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.426014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.438756] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.453014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.459015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.477017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.488213] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.497014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.505000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.519013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.531927] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.546013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.554016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.572016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.582000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.589013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.594014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.609000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.614752] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.620015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.625023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.641012] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.654147] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.656317] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.658404] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.661361] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.663271] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.669044] iommu: Default domain type: Passthrough [ 0.670421] SCSI subsystem initialized [ 0.671108] ACPI: bus type USB registered [ 0.672084] usbcore: registered new interface driver usbfs [ 0.673053] usbcore: registered new interface driver hub [ 0.674063] usbcore: registered new device driver usb [ 0.676148] pps_core: LinuxPPS API ver. 1 registered [ 0.677008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.680053] PTP clock support registered [ 0.682037] EDAC MC: Ver: 3.0.0 [ 0.683455] PCI: Using ACPI for IRQ routing [ 0.684776] NetLabel: Initializing [ 0.686008] NetLabel: domain hash size = 128 [ 0.687007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.689069] NetLabel: unlabeled traffic allowed by default [ 0.690302] vgaarb: loaded [ 0.691235] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.693011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.699878] clocksource: Switched to clocksource kvm-clock [ 0.816636] VFS: Disk quotas dquot_6.6.0 [ 0.817694] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.819675] *** VALIDATE ramfs *** [ 0.820567] *** VALIDATE hugetlbfs *** [ 0.821761] pnp: PnP ACPI init [ 0.823706] pnp: PnP ACPI: found 6 devices [ 0.838149] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.840717] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.842244] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.843853] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.845668] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.847372] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.849929] NET: Registered protocol family 2 [ 0.852241] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.857307] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.861069] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.866166] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.869559] TCP: Hash tables configured (established 65536 bind 65536) [ 0.873820] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.877497] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880673] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.884820] NET: Registered protocol family 1 [ 0.887280] RPC: Registered named UNIX socket transport module. [ 0.889581] RPC: Registered udp transport module. [ 0.891479] RPC: Registered tcp transport module. [ 0.893450] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.896144] NET: Registered protocol family 44 [ 0.897564] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.899534] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.901493] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.904091] PCI: CLS 0 bytes, default 64 [ 0.905977] Unpacking initramfs... [ 2.353540] debug: unmapping init [mem 0xffff9fbefcc54000-0xffff9fbefffbffff] [ 2.357305] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.359186] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.361528] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.854067] Initialise system trusted keyrings [ 2.856021] Key type blacklist registered [ 2.857982] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.867049] zbud: loaded [ 2.869799] *** VALIDATE nfs *** [ 2.870726] *** VALIDATE nfs4 *** [ 2.872285] pstore: using deflate compression [ 2.875795] Platform Keyring initialized [ 2.974921] NET: Registered protocol family 38 [ 2.976836] Key type asymmetric registered [ 2.978248] Asymmetric key parser 'x509' registered [ 2.979933] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.983058] io scheduler mq-deadline registered [ 2.984978] io scheduler kyber registered [ 2.986381] io scheduler bfq registered [ 2.988116] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.991669] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.994671] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.997899] ACPI: Power Button [PWRF] [ 3.003624] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.010389] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.021981] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.028475] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.043496] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.072050] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.099374] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.104383] Non-volatile memory driver v1.3 [ 3.106316] Linux agpgart interface v0.103 [ 3.141498] virtio_blk virtio1: [vda] 155184 512-byte logical blocks (79.5 MB/75.8 MiB) [ 3.144848] vda: detected capacity change from 0 to 79454208 [ 3.160283] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.163565] vdb: detected capacity change from 0 to 1073741824 [ 3.181959] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.185137] vdc: detected capacity change from 0 to 2621440000 [ 3.200892] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.204574] vdd: detected capacity change from 0 to 2621440000 [ 3.222613] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.225920] vde: detected capacity change from 0 to 4294967296 [ 3.241563] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.244337] vdf: detected capacity change from 0 to 4294967296 [ 3.251392] libphy: Fixed MDIO Bus: probed [ 3.256664] usbcore: registered new interface driver usbserial_generic [ 3.259967] usbserial: USB Serial support registered for generic [ 3.261913] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.266256] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.268082] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.270695] mousedev: PS/2 mouse device common for all mice [ 3.275456] rtc_cmos 00:05: RTC can wake from S4 [ 3.279664] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.282853] rtc_cmos 00:05: registered as rtc0 [ 3.291694] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.293821] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.298775] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.300466] intel_pstate: CPU model not supported [ 3.306778] hid: raw HID events driver (C) Jiri Kosina [ 3.309168] usbcore: registered new interface driver usbhid [ 3.311046] usbhid: USB HID core driver [ 3.312255] drop_monitor: Initializing network drop monitor service [ 3.314714] Initializing XFRM netlink socket [ 3.316721] NET: Registered protocol family 10 [ 3.319568] Segment Routing with IPv6 [ 3.320341] NET: Registered protocol family 17 [ 3.322076] mpls_gso: MPLS GSO support [ 3.327720] RAS: Correctable Errors collector initialized. [ 3.330243] AVX version of gcm_enc/dec engaged. [ 3.332186] AES CTR mode by8 optimization enabled [ 3.422458] sched_clock: Marking stable (3422424187, 0)->(4396028382, -973604195) [ 3.425896] registered taskstats version 1 [ 3.427594] Loading compiled-in X.509 certificates [ 3.429632] zswap: loaded using pool lzo/zbud [ 3.455575] Key type big_key registered [ 3.470258] Key type encrypted registered [ 3.471787] ima: No TPM chip found, activating TPM-bypass! [ 3.473419] ima: Allocated hash algorithm: sha1 [ 3.474848] ima: No architecture policies found [ 3.476103] evm: Initialising EVM extended attributes: [ 3.477847] evm: security.selinux [ 3.479037] evm: security.ima [ 3.479963] evm: security.capability [ 3.481022] evm: HMAC attrs: 0x1 [ 3.483299] rtc_cmos 00:05: setting system clock to 2026-09-09 03:57:06 UTC (1788926226) [ 3.489614] debug: unmapping init [mem 0xffffffff9c003000-0xffffffff9c1fffff] [ 3.492469] debug: unmapping init [mem 0xffffffff9ad82000-0xffffffff9b058fff] [ 3.502112] Write protecting the kernel read-only data: 28672k [ 3.505897] debug: unmapping init [mem 0xffffffff99403000-0xffffffff995fffff] [ 3.508579] debug: unmapping init [mem 0xffffffff99d14000-0xffffffff99dfffff] [ 3.543969] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.552578] systemd[1]: Detected virtualization kvm. [ 3.553643] systemd[1]: Detected architecture x86-64. [ 3.555143] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.579667] systemd[1]: No hostname configured. [ 3.581445] systemd[1]: Set hostname to . [ 3.583432] random: systemd: uninitialized urandom read (16 bytes read) [ 3.585726] systemd[1]: Initializing machine ID from random generator. [ 3.739505] random: systemd: uninitialized urandom read (16 bytes read) [ 3.745400] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.750258] random: systemd: uninitialized urandom read (16 bytes read) [ 3.753600] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.763100] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.518720] device-mapper: uevent: version 1.0.3 [ 4.521939] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.347464] virtio_net virtio0 ens2: renamed from eth0 [ 5.349118] random: fast init done [ 5.367905] scsi host0: ata_piix [ 5.386589] scsi host1: ata_piix [ 5.388661] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.392067] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.811533] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.985403] random: crng init done [ 9.986780] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.522326] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.749486] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.008059] SELinux: Disabled at runtime. [ 12.069247] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.082746] systemd[1]: Detected virtualization kvm. [ 12.084821] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.076371] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.093596] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.118946] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.129330] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.137831] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.159889] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.175333] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ 13.764404] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 14.856403] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.114932] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.240408] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.009847] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.131504] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is runni[ 22.145493] Key type dns_resolver registered ng for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 22.993980] NFS: Registering the id_resolver key type [ 22.998387] Key type id_resolver registered [ 23.000690] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg435-server login: [ 51.739044] hrtimer: interrupt took 9078336 ns [ 100.589456] libcfs: loading out-of-tree module taints kernel. [ 100.651485] Key type ._llcrypt registered [ 100.660396] Key type .llcrypt registered [ 100.776984] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 123.626498] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 125.451213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 125.465042] alg: No test for adler32 (adler32-zlib) [ 126.997642] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 128.363757] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 130.487180] Key type lgssc registered [ 132.521969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 150.620438] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 197.669975] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 212.737935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 212.793325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.237224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 214.303995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 214.407106] Lustre: lustre-MDT0000: new disk, initializing [ 214.536098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.573674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 219.668691] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.357459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 233.475819] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 233.500807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 233.504544] Lustre: Skipped 1 previous similar message [ 233.579390] Lustre: lustre-MDT0001: new disk, initializing [ 233.632602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 233.656734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 233.670715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 238.338856] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.471175] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 254.211243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.488809] Lustre: lustre-OST0000: new disk, initializing [ 254.494683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 254.502741] Lustre: 8465:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.580804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.107037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 259.116065] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.236243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 260.974898] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.912267] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 276.012594] Lustre: lustre-OST0001: new disk, initializing [ 276.015564] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 276.020747] Lustre: 9538:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 276.082167] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 282.608813] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 284.727156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 284.743113] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 284.806812] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 295.530059] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 305.619716] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 312.753481] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing check_logdir /tmp/testlogs/ [ 317.690505] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing yml_node [ 323.298684] Lustre: DEBUG MARKER: Client: 2.17.57.55 [ 325.783601] Lustre: DEBUG MARKER: MDS: 2.17.57.55 [ 328.142617] Lustre: DEBUG MARKER: OSS: 2.17.57.55 [ 329.720390] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 9 00:02:31 EDT 2026 [ 346.023258] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 347.433978] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 366.561491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.578055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.584361] Lustre: Skipped 3 previous similar messages [ 366.598328] Lustre: Skipped 2 previous similar messages [ 369.003700] Lustre: server umount lustre-MDT0000 complete [ 372.522683] LustreError: 10412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788926595 with bad export cookie 10906178662890424517 [ 372.534457] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 372.540026] LustreError: 10412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 376.804101] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.806511] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.806714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 376.817391] Lustre: Skipped 1 previous similar message [ 376.835847] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 377.828109] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 377.828661] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 377.861414] Lustre: Skipped 2 previous similar messages [ 379.020647] Lustre: server umount lustre-MDT0001 complete [ 383.185116] Lustre: server umount lustre-OST0000 complete [ 387.471291] Lustre: server umount lustre-OST0001 complete [ 396.080155] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 411.909785] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 452.716893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.970928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 453.031921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 453.125254] Lustre: lustre-MDT0000: new disk, initializing [ 453.235345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.254156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 457.917657] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 468.486382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.644523] Lustre: 15115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 468.726984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 468.733781] Lustre: Skipped 1 previous similar message [ 468.804886] Lustre: lustre-MDT0001: new disk, initializing [ 468.899869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 468.936799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 468.952551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 473.019067] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 481.753449] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 487.459993] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 494.533354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 494.969423] Lustre: lustre-OST0000: new disk, initializing [ 494.977657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 494.985498] Lustre: 17083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 495.069783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.101953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 496.130505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 496.260250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 500.837321] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 512.853883] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 519.580508] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 519.845483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 525.365723] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 525.751787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 531.938763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.944834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 531.966769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 534.696735] Lustre: server umount lustre-OST0000 complete [ 540.642972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 540.650383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.673450] Lustre: Skipped 1 previous similar message [ 540.679363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 540.690435] Lustre: Skipped 1 previous similar message [ 546.633927] Lustre: server umount lustre-MDT0000 complete [ 547.299805] LustreError: 16485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.330098] LustreError: 16485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 550.900245] LustreError: 15107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788926773 with bad export cookie 10906178662890426050 [ 550.914172] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.921350] LustreError: 15107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 551.407176] Lustre: server umount lustre-MDT0001 complete [ 556.584522] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:06:17 (1788926777) [ 559.346819] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 561.601410] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:06:22 (1788926782) [ 569.008276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.253331] LustreError: 19492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.379329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.463776] LustreError: 19493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.262250] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 585.708584] LustreError: 19492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.767486] LustreError: 19492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 590.882144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 597.281058] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 605.631188] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 611.331072] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 618.383944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 625.830567] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 636.800188] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 655.841923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.858327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 655.868913] Lustre: Skipped 2 previous similar messages [ 655.887807] Lustre: Skipped 2 previous similar messages [ 658.409221] Lustre: server umount lustre-OST0000 complete [ 666.081536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.641034] Lustre: server umount lustre-MDT0000 complete [ 671.202117] LustreError: 20221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.219919] LustreError: 20221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 673.611846] LustreError: 19472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788926896 with bad export cookie 10906178662890426932 [ 673.616462] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 673.634299] LustreError: 19472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 673.905589] Lustre: server umount lustre-MDT0001 complete [ 686.450919] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 690.919846] Key type lgssc unregistered [ 691.295223] LNet: 23056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.309920] LNetError: 23056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.336030] LNet: Removed LNI 192.168.204.135@tcp [ 692.768603] Key type .llcrypt unregistered [ 692.770705] Key type ._llcrypt unregistered [ 718.616036] Key type ._llcrypt registered [ 718.617899] Key type .llcrypt registered [ 718.734303] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 720.525320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 720.641247] alg: No test for adler32 (adler32-zlib) [ 721.657556] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 721.912439] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 723.559401] Key type lgssc registered [ 725.746538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 735.592703] loop: module loaded [ 735.689913] loop0: detected capacity change from 0 to 204800000 [ 735.884404] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 735.896767] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 736.269733] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 742.890823] loop0: detected capacity change from 0 to 204800000 [ 743.171131] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 743.201699] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 743.230791] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 749.365856] loop0: detected capacity change from 0 to 204800000 [ 754.917618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 754.937071] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 756.284551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 756.353749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 756.460170] Lustre: lustre-MDT0000: new disk, initializing [ 756.536835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 756.557565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 761.870188] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 771.260122] loop1: detected capacity change from 0 to 204800000 [ 777.228534] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 777.553343] Lustre: lustre-OST07c6: new disk, initializing [ 777.557436] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 777.562138] Lustre: Skipped 1 previous similar message [ 777.572312] Lustre: 26247:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 777.631350] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 784.018447] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 784.029439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 784.040291] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 784.106283] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 789.133108] Lustre: 25236:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 789.141061] Lustre: 25236:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 789.149013] Lustre: 25236:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 789.157040] Lustre: 25236:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 789.162651] Lustre: 25236:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 789.168032] Lustre: 25236:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 804.321177] Lustre: 26914:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 804.334599] Lustre: 26914:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 804.339760] Lustre: 26914:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 804.353517] Lustre: 26914:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 804.359727] Lustre: 26914:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 804.367398] Lustre: 26914:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 809.229871] Lustre: 25236:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 809.246228] Lustre: 25236:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 809.255591] Lustre: 25236:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 809.265435] Lustre: 25236:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 809.273788] Lustre: 25236:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 809.285141] Lustre: 25236:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 809.293390] Lustre: 25236:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 809.302760] Lustre: 25236:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 809.311746] Lustre: 25236:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 809.318411] Lustre: 25236:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 809.323594] Lustre: 25236:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 809.332251] Lustre: 25236:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.579758] Lustre: 25234:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 812.595267] Lustre: 25234:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 812.618471] Lustre: 25234:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 812.642334] Lustre: 25234:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.658603] Lustre: 25234:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 812.672282] Lustre: 25234:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.679827] Lustre: 25234:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 812.693362] Lustre: 25234:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.703487] Lustre: 25234:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 812.717555] Lustre: 25234:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.728291] Lustre: 25234:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 812.749457] Lustre: 25234:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.938644] Lustre: 25234:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 817.956562] Lustre: 25234:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 817.964442] Lustre: 25234:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 817.977378] Lustre: 25234:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.986942] Lustre: 25234:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 818.003861] Lustre: 25234:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.008777] Lustre: 25234:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 818.032145] Lustre: 25234:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.046856] Lustre: 25234:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 818.055572] Lustre: 25234:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.073514] Lustre: 25234:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 818.084189] Lustre: 25234:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 826.707769] Lustre: 25234:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 826.712290] Lustre: 25234:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 826.715691] Lustre: 25234:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 826.719456] Lustre: 25234:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 826.725400] Lustre: 25234:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 826.733890] Lustre: 25234:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 826.739196] Lustre: 25234:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 826.743732] Lustre: 25234:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 826.750307] Lustre: 25234:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 826.756485] Lustre: 25234:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 826.769938] Lustre: 25234:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 826.778914] Lustre: 25234:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 879.012945] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.027047] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 883.828290] Lustre: server umount lustre-OST07c6 complete [ 896.638391] Lustre: server umount lustre-MDT0000 complete [ 916.268457] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 919.491794] Key type lgssc unregistered [ 919.985455] LNet: 28807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.990078] LNetError: 28807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.006109] LNet: Removed LNI 192.168.204.135@tcp [ 920.974437] Key type .llcrypt unregistered [ 920.976843] Key type ._llcrypt unregistered [ 941.140869] Key type ._llcrypt registered [ 941.144457] Key type .llcrypt registered [ 941.277885] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 942.543073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 942.607199] alg: No test for adler32 (adler32-zlib) [ 943.775764] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 944.173901] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 945.919176] Key type lgssc registered [ 947.024815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 955.884942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 955.911634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.678166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 961.900213] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 967.141938] LustreError: 30079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 967.162292] LustreError: 30079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 970.478515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 970.807683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 975.507512] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 984.110025] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 991.232560] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1000.723456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1001.227680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.268930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1009.755186] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1021.293944] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1041.376772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1041.380807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.411864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1041.428922] Lustre: Skipped 1 previous similar message [ 1041.442039] Lustre: Skipped 1 previous similar message [ 1046.075268] Lustre: server umount lustre-OST0000 complete [ 1051.621360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.624407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.646814] Lustre: Skipped 1 previous similar message [ 1056.738294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.747964] Lustre: Skipped 2 previous similar messages [ 1057.235222] Lustre: server umount lustre-MDT0000 complete [ 1061.752597] LustreError: 30061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788927284 with bad export cookie 18298127987885147452 [ 1061.768570] LustreError: 30061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1061.776274] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.240990] Lustre: server umount lustre-MDT0001 complete [ 1077.714184] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1081.628570] Key type lgssc unregistered [ 1082.033109] LNet: 33821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.050580] LNetError: 33821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1082.086027] LNet: Removed LNI 192.168.204.135@tcp [ 1083.114246] Key type .llcrypt unregistered [ 1083.141623] Key type ._llcrypt unregistered [ 1103.099637] Key type ._llcrypt registered [ 1103.103148] Key type .llcrypt registered [ 1103.216686] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1104.287746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1104.398327] alg: No test for adler32 (adler32-zlib) [ 1105.543901] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 1105.859341] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1107.561850] Key type lgssc registered [ 1108.682767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1116.443660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1116.468505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.861813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1122.351779] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1127.399688] LustreError: 35095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.423161] LustreError: 35095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1131.582472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.845116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1136.809446] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1144.706091] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1151.348773] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1159.141520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1159.709278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.770021] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1166.851252] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1174.687793] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1186.276588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1186.280493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1186.289837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1186.316943] Lustre: Skipped 1 previous similar message [ 1190.562347] Lustre: server umount lustre-OST0000 complete [ 1196.515065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.529695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.557589] Lustre: Skipped 1 previous similar message [ 1196.593390] Lustre: Skipped 2 previous similar messages [ 1201.017494] Lustre: server umount lustre-MDT0000 complete [ 1201.642731] LustreError: 35823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.684991] LustreError: 35823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1205.439671] LustreError: 35076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788927428 with bad export cookie 6255961442795053612 [ 1205.446804] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1205.456279] LustreError: 35076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1205.867326] Lustre: server umount lustre-MDT0001 complete [ 1219.166203] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1222.070939] Key type lgssc unregistered [ 1222.402768] LNet: 38631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1222.408071] LNetError: 38631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1222.424808] LNet: Removed LNI 192.168.204.135@tcp [ 1223.459343] Key type .llcrypt unregistered [ 1223.463799] Key type ._llcrypt unregistered [ 1246.950396] Key type ._llcrypt registered [ 1246.953159] Key type .llcrypt registered [ 1247.128349] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1248.505450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1248.583865] alg: No test for adler32 (adler32-zlib) [ 1249.600293] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 1249.869805] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1251.552611] Key type lgssc registered [ 1252.753923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1261.591617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1261.618803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1263.205852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.652975] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1273.844024] LustreError: 40047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.877684] LustreError: 40047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1278.829509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1278.949604] LustreError: 40048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1279.305125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1284.919931] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1292.520106] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1298.701456] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1305.540840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1305.975280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1308.028229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1312.810260] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1321.295667] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1334.533672] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) [ 1335.779399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.785578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1335.786618] Lustre: Skipped 1 previous similar message [ 1335.799923] Lustre: Skipped 1 previous similar message [ 1338.233564] Lustre: server umount lustre-MDT0000 complete [ 1339.630873] LustreError: 40042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.832352] LustreError: 40784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788927565 with bad export cookie 14554353144686430953 [ 1342.842953] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1342.850905] LustreError: 40784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1344.716852] LustreError: 41833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1344.740734] LustreError: 41833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1344.747336] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.35@tcp (stopping) [ 1344.761281] Lustre: Skipped 1 previous similar message [ 1346.017949] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.047280] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1347.049556] Lustre: Skipped 1 previous similar message [ 1349.840452] LustreError: 40044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.857678] LustreError: 40044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1352.162281] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1352.168145] Lustre: Skipped 1 previous similar message [ 1357.279226] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1357.827485] Lustre: server umount lustre-MDT0001 complete [ 1374.525135] Lustre: server umount lustre-OST0000 complete [ 1391.571819] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1394.611153] Key type lgssc unregistered [ 1395.066530] LNet: 43911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1395.071786] LNetError: 43911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1395.089956] LNet: Removed LNI 192.168.204.135@tcp [ 1396.186592] Key type .llcrypt unregistered [ 1396.188371] Key type ._llcrypt unregistered [ 1422.978315] Key type ._llcrypt registered [ 1422.982663] Key type .llcrypt registered [ 1423.075677] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1424.652804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1424.687209] alg: No test for adler32 (adler32-zlib) [ 1425.714463] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 1425.970230] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1427.696121] Key type lgssc registered [ 1428.893890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1436.268770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1436.297843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1437.745311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.227819] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1447.908132] LustreError: 45331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1447.936079] LustreError: 45331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1452.281544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.903930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1457.697875] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1465.646513] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1471.539829] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1479.489818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1479.919298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.037383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1487.427762] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1495.696915] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1503.730881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1503.740601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1503.768834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1507.808288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1507.810986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.823909] Lustre: Skipped 1 previous similar message [ 1509.100629] Lustre: lustre-OST0000: Not available for connect from 192.168.204.35@tcp (stopping) [ 1509.118081] Lustre: Skipped 1 previous similar message [ 1509.413729] Lustre: server umount lustre-OST0000 complete [ 1529.831105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1529.842178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1529.866180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1532.391469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.639645] Lustre: server umount lustre-MDT0000 complete [ 1537.515143] LustreError: 45328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1537.549574] LustreError: 45328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1539.071083] LustreError: 46069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788927762 with bad export cookie 8812464494187779364 [ 1539.075333] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1539.079128] LustreError: 46069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1545.632400] Lustre: server umount lustre-MDT0001 complete [ 1559.129618] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1563.241518] Key type lgssc unregistered [ 1563.682689] LNet: 49051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.699681] LNetError: 49051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1563.727650] LNet: Removed LNI 192.168.204.135@tcp [ 1565.088534] Key type .llcrypt unregistered [ 1565.096205] Key type ._llcrypt unregistered [ 1593.446346] Key type ._llcrypt registered [ 1593.451824] Key type .llcrypt registered [ 1593.562727] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1594.347057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1594.395637] alg: No test for adler32 (adler32-zlib) [ 1595.436867] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 1595.758321] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1597.455322] Key type lgssc registered [ 1598.648507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1607.310543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1607.326610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1608.824800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1614.945590] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1618.404685] LustreError: 50469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1618.418849] LustreError: 50469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1623.532481] LustreError: 50468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.524209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1625.960048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1631.233786] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1638.432323] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1645.010147] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1652.148340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1652.472274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1653.495156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1659.727317] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1670.776924] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1679.395103] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1685.001060] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1699.034851] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-09 0h25m17s [ 1701.489664] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1704.721274] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) [ 1705.803197] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) [ 1706.987391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1706.994511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1707.020863] Lustre: Skipped 1 previous similar message [ 1710.476039] Lustre: server umount lustre-MDT0000 complete [ 1712.332150] LustreError: 50464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.229570] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1714.658637] LustreError: 52609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.684096] LustreError: 52609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1720.810208] LustreError: 50468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1720.838923] LustreError: 50468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1721.834521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.026856] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1722.354183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1722.418855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1723.371036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1727.142742] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1731.469764] Lustre: DEBUG MARKER: Wait for df (36781) ... [ 1733.280907] Lustre: DEBUG MARKER: done [ 1738.732718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.744651] Lustre: Skipped 2 previous similar messages [ 1738.770034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1738.774420] Lustre: Skipped 6 previous similar messages [ 1743.850938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1743.858151] Lustre: Skipped 1 previous similar message [ 1744.204043] Lustre: server umount lustre-OST0000 complete [ 1748.961705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.985597] Lustre: Skipped 1 previous similar message [ 1754.081407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1754.095029] Lustre: Skipped 5 previous similar messages [ 1762.271382] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1762.514747] Lustre: server umount lustre-MDT0000 complete [ 1766.373546] LustreError: 50468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1766.417453] LustreError: 50468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1766.617402] LustreError: 52313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788927989 with bad export cookie 15089341291211899189 [ 1766.618164] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1766.627397] LustreError: 52313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1767.011198] Lustre: server umount lustre-MDT0001 complete [ 1777.944873] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1781.172669] Key type lgssc unregistered [ 1781.573094] LNet: 55399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.581050] LNetError: 55399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.601789] LNet: Removed LNI 192.168.204.135@tcp [ 1782.469644] Key type .llcrypt unregistered [ 1782.472373] Key type ._llcrypt unregistered [ 1831.174388] Key type ._llcrypt registered [ 1831.182314] Key type .llcrypt registered [ 1831.327243] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1833.050146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1833.074261] alg: No test for adler32 (adler32-zlib) [ 1834.260164] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 1834.528568] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1836.191385] Key type lgssc registered [ 1837.705901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1846.508617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1846.526758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1847.745634] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1847.765881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1847.998415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.997801] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1864.229300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.443380] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1864.517285] Lustre: 57618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1864.847061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1870.380144] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1879.254831] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1886.315408] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1894.152704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1894.659461] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1894.993820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1903.912356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1904.047684] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1916.110494] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1923.656854] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-09 0h29m04s [ 1925.910177] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1935.388964] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1938.680532] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1938.736320] LustreError: 57625:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1959.198221] Lustre: lustre-MDT0000: Client 6f21951e-7abd-4ce9-be17-e6874b6d586a (at 192.168.204.35@tcp) reconnecting [ 1978.807189] LustreError: 57625:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1981.836047] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1985.374903] Lustre: DEBUG MARKER: done [ 1996.257036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1996.260141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.268023] LustreError: Skipped 1 previous similar message [ 1996.268627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1996.299846] Lustre: Skipped 1 previous similar message [ 1998.666309] Lustre: server umount lustre-OST0000 complete [ 2006.503907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.521059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2006.523845] Lustre: Skipped 1 previous similar message [ 2006.545672] Lustre: Skipped 2 previous similar messages [ 2007.586363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.597815] Lustre: Skipped 1 previous similar message [ 2010.783538] Lustre: server umount lustre-MDT0000 complete [ 2015.342863] LustreError: 57610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788928238 with bad export cookie 11109783661055463818 [ 2015.346185] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2015.365736] LustreError: 57610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2015.898260] Lustre: server umount lustre-MDT0001 complete [ 2031.285259] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2035.003483] Key type lgssc unregistered [ 2035.664581] LNet: 61955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.683455] LNetError: 61955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.771037] LNet: Removed LNI 192.168.204.135@tcp [ 2037.324690] Key type .llcrypt unregistered [ 2037.333524] Key type ._llcrypt unregistered [ 2091.416362] Key type ._llcrypt registered [ 2091.420871] Key type .llcrypt registered [ 2091.584294] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2093.698137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2093.723642] alg: No test for adler32 (adler32-zlib) [ 2094.953175] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 2095.258094] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2097.048595] Key type lgssc registered [ 2098.309062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2103.465534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2103.494330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.086438] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2105.120185] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2105.285138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2106.298571] Lustre: Failing over lustre-MDT0000 [ 2108.667703] Lustre: server umount lustre-MDT0000 complete [ 2116.339091] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:32:17 (1788928337) [ 2125.539556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.914540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.872260] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2143.291151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.501174] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2143.569397] Lustre: 64755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2144.093108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2150.732206] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2161.968201] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2168.781720] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2176.944725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2177.306219] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2177.546242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.518269] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2188.051803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2194.556744] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2202.051235] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2213.861750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.864589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2213.888606] Lustre: Skipped 1 previous similar message [ 2213.924112] Lustre: Skipped 2 previous similar messages [ 2214.827950] Lustre: server umount lustre-MDT0000 complete [ 2220.591429] LustreError: 64749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788928443 with bad export cookie 18052179386082756260 [ 2220.594954] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2220.619517] LustreError: 64749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2221.229583] Lustre: server umount lustre-MDT0001 complete [ 2225.279492] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2230.663751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2240.480044] Lustre: 63680:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788928447/real 1788928447] req@00000000d07ffb57 x1875827300020480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788928463 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2240.506525] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.520133] Lustre: Skipped 1 previous similar message [ 2240.561249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.503753] Lustre: 63678:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788928448/real 1788928448] req@00000000e97f9154 x1875827300020608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788928464 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2245.663351] Lustre: 63678:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788928452/real 1788928452] req@000000003e386b5d x1875827300020736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788928468 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2246.130566] LustreError: 66611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2246.171068] LustreError: 66611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2246.326744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2246.457475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2249.315454] LustreError: 66612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.342927] LustreError: 68532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.371053] LustreError: 68532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2251.688264] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2252.256314] Lustre: 63680:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788928458/real 1788928458] req@000000007912d1ae x1875827300021120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788928474 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2252.278165] Lustre: 63680:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2255.840666] LustreError: 68533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.863666] LustreError: 68533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2256.480957] Lustre: 63679:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788928463/real 1788928463] req@000000006f39d031 x1875827300021248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788928479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2260.763619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.966503] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2260.977148] Lustre: Skipped 1 previous similar message [ 2261.162027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2266.608123] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2266.671714] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2269.738777] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2274.986499] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2281.341502] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2290.145889] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.157951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.177515] Lustre: Skipped 1 previous similar message [ 2290.185740] Lustre: Skipped 2 previous similar messages [ 2294.363876] Lustre: server umount lustre-MDT0000 complete [ 2295.279510] LustreError: 69974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.307508] LustreError: 69974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2298.316036] LustreError: 65493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788928521 with bad export cookie 18052179386082758535 [ 2298.318584] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2298.325133] LustreError: 65493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2298.643376] Lustre: server umount lustre-MDT0001 complete [ 2306.231075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2316.769076] Lustre: 63681:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788928523/real 1788928523] req@000000004fbd2fb6 x1875827300049536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788928539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2316.804310] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.821128] Lustre: Skipped 1 previous similar message [ 2318.259032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.000416] LustreError: 63677:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004fbd2fb6 x1875827300050560/t0(0) o250->MGC192.168.204.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2324.371450] LustreError: 66612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.482260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2324.559596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2329.231134] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2339.299061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.940610] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2344.964518] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2354.075408] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2361.303115] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2371.558465] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.563037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.582473] Lustre: Skipped 2 previous similar messages [ 2371.592047] Lustre: Skipped 2 previous similar messages [ 2376.673042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.685764] Lustre: Skipped 2 previous similar messages [ 2385.376515] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2385.991770] Lustre: server umount lustre-MDT0000 complete [ 2386.915574] LustreError: 71216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.940565] LustreError: 71216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2389.715634] LustreError: 64748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788928612 with bad export cookie 18052179386082759914 [ 2389.716404] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2389.734404] LustreError: 64748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2390.001962] Lustre: server umount lustre-MDT0001 complete [ 2397.021122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2400.466991] Lustre: DEBUG MARKER: files compared the same [ 2413.217220] Lustre: server umount lustre-OST0000 complete [ 2432.550578] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2435.848269] Key type lgssc unregistered [ 2436.236739] LNet: 74500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2436.247110] LNetError: 74500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2436.270276] LNet: Removed LNI 192.168.204.135@tcp [ 2437.170551] Key type .llcrypt unregistered [ 2437.172405] Key type ._llcrypt unregistered [ 2462.201517] Key type ._llcrypt registered [ 2462.203600] Key type .llcrypt registered [ 2462.354463] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2463.735619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2463.773591] alg: No test for adler32 (adler32-zlib) [ 2464.894329] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 2465.168127] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2466.880089] Key type lgssc registered [ 2468.186336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2476.936984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2476.951828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2478.453222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.889420] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug +malloc all [ 2488.802252] LustreError: 75917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.822724] LustreError: 75917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2493.921932] LustreError: 75918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2494.608974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.061295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2501.367806] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug +malloc all [ 2513.062260] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2520.610839] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2529.245430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2529.696922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2536.868894] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug +malloc all [ 2546.579085] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2548.178337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2558.432625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.436965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2558.450852] Lustre: Skipped 1 previous similar message [ 2558.451053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2558.477575] Lustre: Skipped 1 previous similar message [ 2563.573391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2563.576632] Lustre: Skipped 1 previous similar message [ 2564.290605] Lustre: server umount lustre-OST0000 complete [ 2568.672547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2568.683975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.701030] Lustre: Skipped 1 previous similar message [ 2568.716083] Lustre: Skipped 1 previous similar message [ 2573.794811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.811464] Lustre: Skipped 1 previous similar message [ 2574.745845] Lustre: server umount lustre-MDT0000 complete [ 2578.914364] LustreError: 78407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.935515] LustreError: 78407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2579.020199] LustreError: 78496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788928802 with bad export cookie 3955023070877694832 [ 2579.023969] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2579.049687] LustreError: 78496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2579.617584] Lustre: server umount lustre-MDT0001 complete [ 2594.968880] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2598.927517] Key type lgssc unregistered [ 2599.222832] LNet: 79453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2599.229980] LNetError: 79453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2599.261609] LNet: Removed LNI 192.168.204.135@tcp [ 2600.341115] Key type .llcrypt unregistered [ 2600.347903] Key type ._llcrypt unregistered [ 2628.107423] Key type ._llcrypt registered [ 2628.112984] Key type .llcrypt registered [ 2628.248251] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2629.822409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2629.869595] alg: No test for adler32 (adler32-zlib) [ 2630.909190] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 2631.180490] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2632.919205] Key type lgssc registered [ 2634.426638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2642.685074] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2642.720345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2658.399535] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2663.521483] LustreError: 80847:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 2690.207518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.702473] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2708.391502] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2717.743682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2718.076908] Lustre: *** cfs_fail_loc=706, val=0*** [ 2718.701611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2718.829575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2723.811200] LustreError: 80872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.839060] LustreError: 80872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2724.390520] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2724.840745] LustreError: 80873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.854155] LustreError: 80873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2729.961840] LustreError: 80872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2729.979491] LustreError: 80872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2734.783444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2735.161316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2740.084562] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2748.593477] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2754.571568] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2759.662182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2759.669237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.690241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2760.674688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2760.680788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2760.693344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.720740] Lustre: Skipped 1 previous similar message [ 2763.089417] Lustre: server umount lustre-OST0000 complete [ 2770.919275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.924537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2770.929805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2770.943301] Lustre: Skipped 1 previous similar message [ 2773.548660] Lustre: server umount lustre-MDT0000 complete [ 2776.032318] LustreError: 81939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.062681] LustreError: 81939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2777.673692] LustreError: 80855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788929000 with bad export cookie 10820124343038599106 [ 2777.693871] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2778.053347] Lustre: server umount lustre-MDT0001 complete [ 2791.161760] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2794.748722] Key type lgssc unregistered [ 2795.176642] LNet: 84472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2795.187770] LNetError: 84472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2795.203574] LNet: Removed LNI 192.168.204.135@tcp [ 2796.231237] Key type .llcrypt unregistered [ 2796.233095] Key type ._llcrypt unregistered [ 2824.921674] Key type ._llcrypt registered [ 2824.923959] Key type .llcrypt registered [ 2825.047897] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2826.174128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2826.268272] alg: No test for adler32 (adler32-zlib) [ 2827.406691] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 2827.776310] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2829.511218] Key type lgssc registered [ 2831.003220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2841.839117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2841.876843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.439711] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2857.629729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2858.223073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2863.586519] LustreError: 86578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.619907] LustreError: 86578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2864.487851] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2868.708893] LustreError: 86582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2868.754118] LustreError: 86582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2873.837075] LustreError: 86581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.860201] LustreError: 86581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2876.192439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2877.066155] LustreError: 86578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.117737] LustreError: 86578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2877.152543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.528522] LustreError: 86582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.547311] LustreError: 86582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2886.737663] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2893.794138] LustreError: 86581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.841679] LustreError: 86581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2898.772356] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2898.807216] Lustre: MGS: Client 6839a1d2-0087-4a2d-abc4-3483d1ae4f57 (at 0@lo) reconnecting [ 2898.832420] LustreError: 86592:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2899.233972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.368144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2904.303710] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2921.955364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2921.967524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.986990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2925.025820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2925.027983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2925.040365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2925.080269] Lustre: Skipped 1 previous similar message [ 2927.350779] Lustre: server umount lustre-OST0000 complete [ 2935.265990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2935.267297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2935.268650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2935.281098] Lustre: Skipped 1 previous similar message [ 2938.281752] Lustre: server umount lustre-MDT0000 complete [ 2940.386406] LustreError: 88122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.405462] LustreError: 88122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2943.085745] Lustre: server umount lustre-MDT0001 complete [ 2948.106407] Lustre: server umount MGS complete [ 2962.456569] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:46:23 (1788929183) [ 2996.863658] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 3008.724308] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3051.810063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.842176] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3067.116557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3067.439860] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3067.560610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3067.713340] Lustre: lustre-MDT0001: new disk, initializing [ 3067.851784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3067.881728] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3067.894632] Lustre: 93780:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3067.910987] LustreError: 93780:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3072.823302] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3084.590464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3084.835511] Lustre: lustre-OST0000: new disk, initializing [ 3084.838818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3084.843949] Lustre: 94545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3084.920813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.090627] Lustre: 94840:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000ba9a032 x1875828067577728/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788929332 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3093.320088] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3103.418369] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3111.833790] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3111.852577] Lustre: MGS: Client 7057a4a0-3d30-4278-8069-347ab9165816 (at 0@lo) reconnecting [ 3111.895827] LustreError: 93766:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3111.917560] Lustre: 93047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3111.981188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3112.060832] Lustre: lustre-MDT0000: new disk, initializing [ 3112.303720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3112.336951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3112.376609] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3113.442962] Lustre: 95614:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008db6e49f x1875828067585024/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788929352 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3117.442404] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3118.516504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3118.527268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3118.608595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3143.654363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.655854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3143.664753] Lustre: Skipped 1 previous similar message [ 3143.668050] Lustre: Skipped 1 previous similar message [ 3148.769859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3148.799615] Lustre: Skipped 2 previous similar messages [ 3149.220930] Lustre: server umount lustre-OST0000 complete [ 3153.898567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3153.899393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.921687] Lustre: Skipped 1 previous similar message [ 3159.008773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3159.018355] Lustre: Skipped 3 previous similar messages [ 3159.340534] Lustre: server umount lustre-MDT0000 complete [ 3170.341847] Lustre: server umount lustre-MDT0001 complete [ 3176.257165] Lustre: server umount MGS complete [ 3190.855947] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:50:11 (1788929411) [ 3212.291325] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3216.146935] Key type lgssc unregistered [ 3216.549859] LNet: 98117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3216.555425] LNetError: 98117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3216.572922] LNet: Removed LNI 192.168.204.135@tcp [ 3217.607116] Key type .llcrypt unregistered [ 3217.609389] Key type ._llcrypt unregistered [ 3237.869700] Key type ._llcrypt registered [ 3237.872540] Key type .llcrypt registered [ 3238.011991] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3239.093653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3239.122981] alg: No test for adler32 (adler32-zlib) [ 3240.189927] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 3240.466804] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3242.120788] Key type lgssc registered [ 3243.500965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3252.697568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3252.739311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.437536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3259.246607] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3265.009427] LustreError: 99340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3265.023267] LustreError: 99340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3269.268405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.627808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3274.269413] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3282.507802] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3288.146130] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3296.359485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3296.885979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3300.981530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3303.335978] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3312.958605] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3321.824417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3321.837087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3321.867117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3324.905419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3324.906847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3324.910563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.963599] Lustre: Skipped 1 previous similar message [ 3327.567984] Lustre: server umount lustre-OST0000 complete [ 3335.140635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3335.157261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3335.164515] Lustre: Skipped 1 previous similar message [ 3338.759700] Lustre: server umount lustre-MDT0000 complete [ 3340.263322] LustreError: 100549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3340.285619] LustreError: 100549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3342.903032] LustreError: 99322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788929565 with bad export cookie 12892561567989722463 [ 3342.907818] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3342.917621] LustreError: 99322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3343.221821] Lustre: server umount lustre-MDT0001 complete [ 3355.505252] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3359.024538] Key type lgssc unregistered [ 3359.390316] LNet: 102878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3359.400619] LNetError: 102878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3359.418865] LNet: Removed LNI 192.168.204.135@tcp [ 3360.334563] Key type .llcrypt unregistered [ 3360.342333] Key type ._llcrypt unregistered [ 3379.765543] Key type ._llcrypt registered [ 3379.767218] Key type .llcrypt registered [ 3379.977799] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3381.128143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3381.204944] alg: No test for adler32 (adler32-zlib) [ 3382.566757] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 3382.957519] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3384.783482] Key type lgssc registered [ 3386.176359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3393.327430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3393.336836] LustreError: 103954:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3393.460856] LustreError: 103965:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3393.475023] LustreError: 103954:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4870 [ 3393.490756] LustreError: 103954:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3393.507530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3393.511312] LustreError: 103954:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3395.136895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.191237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3404.537844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3409.918574] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3417.070915] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing lsmod [ 3420.301346] LustreError: 105172:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3420.304142] LustreError: 105171:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3420.318985] LustreError: 105172:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3420.331132] LustreError: 105172:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3420.350353] LustreError: 105172:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3420.363201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.797175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.829883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3425.248891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3425.258605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3425.277969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.297193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.298210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.314551] Lustre: Skipped 1 previous similar message [ 3427.332310] Lustre: Skipped 2 previous similar messages [ 3429.362243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3429.371398] Lustre: Skipped 2 previous similar messages [ 3430.424551] Lustre: server umount lustre-MDT0000 complete [ 3434.473228] LustreError: 104003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3434.499795] LustreError: 104003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3434.521791] LustreError: 103977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788929657 with bad export cookie 13249958761481020008 [ 3434.533946] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3434.545161] LustreError: 103977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3435.554123] LustreError: 104434:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000744795b2 x1875828649515392/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3435.732510] Lustre: server umount lustre-MDT0001 complete [ 3450.805019] Lustre: server umount lustre-OST0000 complete [ 3461.080568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.743469] LustreError: 106237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.845275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3467.237064] LustreError: 106238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3467.473790] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3471.333174] LustreError: 106237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3476.459309] LustreError: 106238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3477.374299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3477.767668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3482.720988] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3491.908883] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3498.797766] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3506.803713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3507.313481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.622943] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3525.136679] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3531.978053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3538.400757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3538.422113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.441497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3540.085601] Lustre: server umount lustre-OST0000 complete [ 3547.631836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3547.656534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3547.659918] Lustre: Skipped 1 previous similar message [ 3547.718820] Lustre: Skipped 1 previous similar message [ 3557.859280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3557.868054] Lustre: Skipped 3 previous similar messages [ 3559.391257] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3559.569527] Lustre: server umount lustre-MDT0000 complete [ 3562.981488] LustreError: 108821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3563.019675] LustreError: 108821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3564.119971] LustreError: 106218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788929787 with bad export cookie 13249958761481020953 [ 3564.128995] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3564.135331] LustreError: 106218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3564.601172] Lustre: server umount lustre-MDT0001 complete [ 3577.707572] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3580.924285] Key type lgssc unregistered [ 3581.373478] LNet: 109798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3581.389323] LNetError: 109798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3581.410464] LNet: Removed LNI 192.168.204.135@tcp [ 3582.607113] Key type .llcrypt unregistered [ 3582.613144] Key type ._llcrypt unregistered [ 3609.344331] Key type ._llcrypt registered [ 3609.347513] Key type .llcrypt registered [ 3609.435699] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3611.403060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3611.462241] alg: No test for adler32 (adler32-zlib) [ 3612.862337] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 3613.301605] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3615.135271] Key type lgssc registered [ 3616.811441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3625.719388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3625.737302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.342574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.481850] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3637.740335] LustreError: 111223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3637.766093] LustreError: 111223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3642.771930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3642.851846] LustreError: 111224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3643.317344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3648.715873] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3656.439571] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3662.444200] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3670.303864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3670.650178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.697522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3678.039781] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3688.248357] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3695.403569] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3699.294563] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3708.900434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3708.913478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3708.953708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3708.977964] Lustre: Skipped 1 previous similar message [ 3711.534362] Lustre: server umount lustre-OST0000 complete [ 3719.141962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.159102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3719.172896] Lustre: Skipped 2 previous similar messages [ 3724.259051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3724.264378] Lustre: Skipped 2 previous similar messages [ 3724.862849] Lustre: server umount lustre-MDT0000 complete [ 3729.391217] LustreError: 111224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3729.453649] LustreError: 111224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3731.667519] LustreError: 113755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788929954 with bad export cookie 9291772731119641925 [ 3731.677616] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3731.692934] LustreError: 113755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3732.121674] Lustre: server umount lustre-MDT0001 complete [ 3748.633743] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3752.246952] Key type lgssc unregistered [ 3752.573612] LNet: 114867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3752.582489] LNetError: 114867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3752.597815] LNet: Removed LNI 192.168.204.135@tcp [ 3753.884189] Key type .llcrypt unregistered [ 3753.888592] Key type ._llcrypt unregistered [ 3773.911486] Key type ._llcrypt registered [ 3773.916612] Key type .llcrypt registered [ 3774.012333] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3774.958664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3774.982266] alg: No test for adler32 (adler32-zlib) [ 3776.224114] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 3776.614107] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3778.319583] Key type lgssc registered [ 3779.612612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3788.447441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3788.494132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.150643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3795.820124] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3800.546510] LustreError: 116090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3800.558261] LustreError: 116090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3805.672894] LustreError: 116089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3806.230610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.849954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3812.055666] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3820.984956] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3826.831689] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3834.143345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3834.643346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3835.641628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3842.519485] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3850.976638] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3861.472280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3861.480878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.508892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3862.501042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3862.506728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3862.531953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3866.274527] Lustre: server umount lustre-OST0000 complete [ 3872.738123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3872.745775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3872.759379] Lustre: Skipped 1 previous similar message [ 3872.763840] Lustre: Skipped 1 previous similar message [ 3877.462736] Lustre: server umount lustre-MDT0000 complete [ 3877.863987] LustreError: 116090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3877.887162] LustreError: 116090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3881.645348] LustreError: 116068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788930104 with bad export cookie 10090479959878671387 [ 3881.652887] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3881.660951] LustreError: 116068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3882.238851] Lustre: server umount lustre-MDT0001 complete [ 3894.695792] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3897.531913] Key type lgssc unregistered [ 3897.906222] LNet: 119636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3897.911203] LNetError: 119636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3897.938370] LNet: Removed LNI 192.168.204.135@tcp [ 3898.887441] Key type .llcrypt unregistered [ 3898.890967] Key type ._llcrypt unregistered [ 3927.966179] Key type ._llcrypt registered [ 3927.973730] Key type .llcrypt registered [ 3928.179952] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3930.091504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3930.127940] alg: No test for adler32 (adler32-zlib) [ 3931.225163] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 3931.571178] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3933.264184] Key type lgssc registered [ 3934.797701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3943.581437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3943.632946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3945.255122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3950.169045] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3955.681851] LustreError: 121058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3955.698563] LustreError: 121058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3959.459464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3960.016044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3964.893386] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3972.574123] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3979.487450] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3987.675875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3988.146377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3990.198289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3995.077542] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4002.703314] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4010.351415] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4015.581353] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4015.591984] Lustre: Skipped 2 previous similar messages [ 4016.793849] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4016.827564] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4020.087511] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4020.098434] Lustre: Skipped 2 previous similar messages [ 4026.507555] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4026.512522] Lustre: Skipped 1 previous similar message [ 4031.674541] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.35@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4031.682434] Lustre: Skipped 2 previous similar messages [ 4032.985428] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.35@tcp 192.168.0.[2,10]@tcp [ 4040.236952] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.35/24@tcp in log lustre-MDT0000 [ 4040.243159] Lustre: Skipped 2 previous similar messages [ 4042.707382] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.35/24@tcp [ 4042.722580] Lustre: Skipped 1 previous similar message [ 4050.914084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4050.919267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4050.932080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4050.943502] Lustre: Skipped 1 previous similar message [ 4050.963540] Lustre: Skipped 1 previous similar message [ 4056.117488] Lustre: server umount lustre-OST0000 complete [ 4061.157278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.177846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4061.187554] Lustre: Skipped 1 previous similar message [ 4066.275270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4066.285352] Lustre: Skipped 1 previous similar message [ 4067.012161] Lustre: server umount lustre-MDT0000 complete [ 4071.337531] LustreError: 123687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788930294 with bad export cookie 9052602792914438671 [ 4071.341516] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4071.348624] LustreError: 123687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4072.052450] Lustre: server umount lustre-MDT0001 complete [ 4084.603352] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 4087.893716] Key type lgssc unregistered [ 4088.168728] LNet: 125864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4088.182664] LNetError: 125864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4088.199501] LNet: Removed LNI 192.168.204.135@tcp [ 4089.103450] Key type .llcrypt unregistered [ 4089.105573] Key type ._llcrypt unregistered [ 4124.189814] Key type ._llcrypt registered [ 4124.192577] Key type .llcrypt registered [ 4124.383558] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 4126.075157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4126.101951] alg: No test for adler32 (adler32-zlib) [ 4127.213244] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 4127.549237] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 4129.231342] Key type lgssc registered [ 4130.376497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4138.686615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4138.721068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4140.350945] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4140.382321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4140.396078] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.35/24@tcp [ 4145.681921] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4150.753793] LustreError: 127426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.775937] LustreError: 127426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4155.019850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4155.498226] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4155.550681] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.35/24@tcp [ 4155.620433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4161.241872] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4169.706308] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4176.640828] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4184.801120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4185.293804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4186.333718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4192.650508] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4201.163856] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4211.174376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4211.176479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4211.190632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4213.205914] Lustre: server umount lustre-OST0000 complete [ 4221.412039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.418981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4221.452736] Lustre: Skipped 2 previous similar messages [ 4221.465645] Lustre: Skipped 2 previous similar messages [ 4224.581338] Lustre: server umount lustre-MDT0000 complete [ 4226.545503] LustreError: 127427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4226.576438] LustreError: 127427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4228.997409] LustreError: 127407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788930452 with bad export cookie 13701457443781596786 [ 4229.008941] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4229.411609] Lustre: server umount lustre-MDT0001 complete [ 4241.599854] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 4245.066106] Key type lgssc unregistered [ 4245.467252] LNet: 131020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4245.480293] LNetError: 131020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4245.509270] LNet: Removed LNI 192.168.204.135@tcp [ 4246.726462] Key type .llcrypt unregistered [ 4246.730142] Key type ._llcrypt unregistered [ 4278.129770] Key type ._llcrypt registered [ 4278.134031] Key type .llcrypt registered [ 4278.346489] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 4302.157261] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 4304.021287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4304.052892] alg: No test for adler32 (adler32-zlib) [ 4305.261705] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 4305.648544] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 4307.431327] Key type lgssc registered [ 4308.906430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4348.658128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4348.688331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4350.182298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4350.249448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4350.365151] Lustre: lustre-MDT0000: new disk, initializing [ 4350.498356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4350.520582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4355.297873] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4366.661287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.834210] Lustre: 134428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4366.941229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4366.950207] Lustre: Skipped 1 previous similar message [ 4367.070725] Lustre: lustre-MDT0001: new disk, initializing [ 4367.232520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4367.288932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4367.303503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4372.166078] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4383.055724] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4389.471847] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4396.010760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4396.323814] Lustre: lustre-OST0000: new disk, initializing [ 4396.331752] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4396.342275] Lustre: 136396:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4396.446994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4403.286881] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4404.303132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4404.320058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4404.501186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4413.424547] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4418.299327] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4418.536992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4423.148916] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4423.322289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4425.195336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4425.196190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4425.204146] LustreError: Skipped 1 previous similar message [ 4425.204743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4425.227019] Lustre: Skipped 1 previous similar message [ 4430.307970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4430.317339] Lustre: Skipped 2 previous similar messages [ 4431.557569] Lustre: server umount lustre-OST0000 complete [ 4439.020913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4439.031352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4439.047341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4440.544151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.666610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4445.682187] Lustre: Skipped 3 previous similar messages [ 4450.272928] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4450.633370] Lustre: server umount lustre-MDT0000 complete [ 4455.011815] LustreError: 134421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788930678 with bad export cookie 10341408921666554380 [ 4455.020741] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.029125] LustreError: 134421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4455.437774] Lustre: server umount lustre-MDT0001 complete [ 4465.021699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4465.739095] LustreError: 138614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4465.763530] LustreError: 138614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4465.888632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4471.275839] LustreError: 138615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4471.798356] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4475.369167] LustreError: 138614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4480.482897] LustreError: 138615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4481.117569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4481.436980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4486.335771] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4494.131763] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4501.083407] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4509.518404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4509.993897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4517.684140] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4526.633338] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4532.234263] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4532.453265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4537.967073] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4538.322700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4549.713509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4549.962681] Lustre: lustre-OST0001: new disk, initializing [ 4549.979498] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4550.015743] Lustre: 141989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4550.161431] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4556.899937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4556.908211] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4557.015845] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4557.719554] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4566.900789] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4567.137711] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4572.145856] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4572.468763] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4579.359205] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b4543700000.ost_server_uuid 50 [ 4581.363836] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b4543700000.ost_server_uuid in IDLE state after 0 sec [ 4593.122780] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.127525] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4593.146213] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4593.159359] Lustre: Skipped 1 previous similar message [ 4597.706599] Lustre: server umount lustre-OST0001 complete [ 4598.251202] LustreError: 140438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4598.271307] LustreError: 140438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4603.366752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4603.369964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4603.383303] LustreError: Skipped 1 previous similar message [ 4603.389597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4603.430623] Lustre: Skipped 2 previous similar messages [ 4608.496746] Lustre: server umount lustre-OST0000 complete [ 4613.601981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4619.600628] Lustre: server umount lustre-MDT0000 complete [ 4623.696983] LustreError: 140459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788930846 with bad export cookie 10341408921666555262 [ 4623.715841] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4623.718406] LustreError: 140459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4624.265193] Lustre: server umount lustre-MDT0001 complete [ 4645.924360] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 4649.218838] Key type lgssc unregistered [ 4649.547059] LNet: 145000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4649.556412] LNetError: 145000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4650.606193] LNet: Removed LNI 192.168.204.135@tcp [ 4651.466162] Key type .llcrypt unregistered [ 4651.471330] Key type ._llcrypt unregistered [ 4698.727371] Key type ._llcrypt registered [ 4698.729528] Key type .llcrypt registered [ 4698.876708] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 4714.304389] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 4715.992570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4716.082144] alg: No test for adler32 (adler32-zlib) [ 4717.166681] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 4717.545637] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 4719.247429] Key type lgssc registered [ 4720.747594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4761.736236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4761.769246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4763.242527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4763.308315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4763.413749] Lustre: lustre-MDT0000: new disk, initializing [ 4763.627441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4763.653597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4768.816948] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4781.112047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4781.248964] Lustre: 149159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4781.347208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4781.352925] Lustre: Skipped 1 previous similar message [ 4781.547865] Lustre: lustre-MDT0001: new disk, initializing [ 4781.696639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4781.776156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4781.786578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4788.029669] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4800.805273] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4808.640088] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4818.997043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4819.381454] Lustre: lustre-OST0000: new disk, initializing [ 4819.384781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4819.392423] Lustre: 151129:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4819.520729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4825.260218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4825.293487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4825.553129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4830.076319] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4842.787138] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4847.653105] Lustre: Failing over lustre-OST0000 [ 4847.988034] Lustre: server umount lustre-OST0000 complete [ 4865.503358] Lustre: 147382:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788931072/real 1788931072] req@0000000067621aff x1875830049379968/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788931088 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4865.541437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4867.551574] Lustre: 147383:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788931074/real 1788931074] req@00000000bef17481 x1875830049380992/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788931090 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4867.591695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4868.968213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4869.482486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4869.501406] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4872.543280] Lustre: 147380:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788931079/real 1788931079] req@00000000412fde4b x1875830049382144/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788931095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4872.605820] Lustre: 147380:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4875.494065] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4877.062556] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4877.669072] Lustre: 147382:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788931084/real 1788931084] req@000000002e0e88d5 x1875830049383424/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788931100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4877.704025] Lustre: 147382:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4877.710630] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4877.710700] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4881.801646] Lustre: Failing over lustre-MDT0000 [ 4882.243637] Lustre: server umount lustre-MDT0000 complete [ 4882.847319] Lustre: 147380:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788931088/real 1788931088] req@000000004f2f4d14 x1875830049384320/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788931104 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4882.894861] Lustre: 147380:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4883.937902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4883.945666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.984604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4886.012033] Lustre: Skipped 1 previous similar message [ 4890.828416] LustreError: 149165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4890.839954] LustreError: 149165:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4895.972522] LustreError: 149166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4895.997965] LustreError: 149166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4901.069103] LustreError: 149164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4901.097651] LustreError: 149164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4902.239140] Lustre: 147380:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788931109/real 1788931109] req@0000000047b65ee4 x1875830049397376/t0(0) o400->MGC192.168.204.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788931125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4902.300859] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4903.947238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4906.187355] LustreError: 151656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4906.218565] LustreError: 151656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4911.328154] LustreError: 149164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4911.348047] LustreError: 149164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4912.615110] LustreError: 147379:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f59761fa x1875830049403392/t0(0) o250->MGC192.168.204.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4912.988693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4913.064224] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4916.429272] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4918.266484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4918.276173] Lustre: Skipped 1 previous similar message [ 4918.305419] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4918.352489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4918.502762] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4933.610305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.627857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4933.628430] Lustre: Skipped 1 previous similar message [ 4933.638213] Lustre: Skipped 1 previous similar message [ 4936.171229] Lustre: server umount lustre-OST0000 complete [ 4943.842712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4943.844784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4943.858938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4943.884843] Lustre: Skipped 1 previous similar message [ 4943.903718] Lustre: Skipped 1 previous similar message [ 4946.550214] Lustre: server umount lustre-MDT0000 complete [ 4948.976627] LustreError: 151656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4949.017067] LustreError: 151656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4951.456284] LustreError: 149150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788931174 with bad export cookie 18349029488281709373 [ 4951.464115] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4951.480320] LustreError: 149150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4952.198487] Lustre: server umount lustre-MDT0001 complete [ 4967.993441] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 4970.716775] Key type lgssc unregistered [ 4971.047447] LNet: 154798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4971.059368] LNetError: 154798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4971.082237] LNet: Removed LNI 192.168.204.135@tcp [ 4972.088200] Key type .llcrypt unregistered [ 4972.090757] Key type ._llcrypt unregistered [ 5015.235496] Key type ._llcrypt registered [ 5015.240413] Key type .llcrypt registered [ 5015.397500] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5017.040817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5017.181359] alg: No test for adler32 (adler32-zlib) [ 5018.375568] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 5018.700026] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5020.392541] Key type lgssc registered [ 5021.901223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5032.753968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5032.777866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5034.554770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5040.185358] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5044.707441] LustreError: 156324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5044.730827] LustreError: 156324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5049.828317] LustreError: 156323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5052.215496] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5054.960810] LustreError: 156324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5056.947160] LustreError: 156352:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5056.958326] LustreError: 156352:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5056.967949] LustreError: 156352:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 5057.477549] Lustre: server umount lustre-MDT0000 complete [ 5076.664908] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 5088.337158] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5130.532709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.981740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5131.055157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5131.202172] Lustre: lustre-MDT0000: new disk, initializing [ 5131.320148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5131.359382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5137.232750] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5150.275302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5150.474339] Lustre: 159658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5150.591889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5150.604659] Lustre: Skipped 1 previous similar message [ 5150.725823] Lustre: lustre-MDT0001: new disk, initializing [ 5150.917212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5150.977402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5150.996219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5156.370051] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5167.851213] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5174.342684] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5182.448562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5182.680382] Lustre: lustre-OST0000: new disk, initializing [ 5182.684527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5182.689478] Lustre: 161626:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5182.794810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5189.239127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5189.254995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5189.423910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5190.341644] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5202.073634] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5206.326127] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5208.550748] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:23:49 (1788931429) [ 5463.520299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5463.527365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5463.541627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5466.080872] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5466.086228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5466.114764] Lustre: Skipped 2 previous similar messages [ 5471.200490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5471.205233] Lustre: Skipped 2 previous similar messages [ 5476.320634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5476.335955] Lustre: Skipped 2 previous similar messages [ 5477.343370] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5477.753822] Lustre: server umount lustre-MDT0000 complete [ 5481.222324] LustreError: 159650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788931704 with bad export cookie 7221316415359178132 [ 5481.229783] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5481.235180] LustreError: 159650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5481.439600] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5481.442430] LustreError: 159665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5481.457709] Lustre: Skipped 1 previous similar message [ 5481.458853] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5486.559927] LustreError: 161885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5487.814942] Lustre: server umount lustre-MDT0001 complete [ 5506.015191] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5506.162254] Lustre: server umount lustre-OST0000 complete [ 5514.380445] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 5522.646028] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5553.394853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.637160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5553.666834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5553.756716] Lustre: lustre-MDT0000: new disk, initializing [ 5553.844243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5553.856562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5557.922983] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5567.355824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5567.459834] Lustre: 166293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5567.518678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5567.525246] Lustre: Skipped 1 previous similar message [ 5567.587502] Lustre: lustre-MDT0001: new disk, initializing [ 5567.676796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5567.707446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5567.713896] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5571.848304] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5579.928383] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5584.758745] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5591.655745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5591.904644] Lustre: lustre-OST0000: new disk, initializing [ 5591.910289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5591.920736] Lustre: 168261:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5592.012512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5598.286679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5598.296324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5598.413601] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5598.446127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5607.208573] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5612.079758] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5612.327561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5616.690745] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5616.893657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5619.188149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5619.188921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.191707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5619.191714] Lustre: Skipped 1 previous similar message [ 5619.201436] LustreError: Skipped 1 previous similar message [ 5619.219709] Lustre: Skipped 1 previous similar message [ 5625.066579] Lustre: server umount lustre-OST0000 complete [ 5629.422488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5629.428221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5634.853581] Lustre: server umount lustre-MDT0000 complete [ 5638.085284] LustreError: 166287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788931861 with bad export cookie 7221316415359336962 [ 5638.092227] LustreError: 166287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5638.097283] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5638.390750] Lustre: server umount lustre-MDT0001 complete [ 5645.035355] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:31:06 (1788931866) [ 5658.928540] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 5667.839244] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5700.342158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5700.551597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5700.571896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5700.635215] Lustre: lustre-MDT0000: new disk, initializing [ 5700.697195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5700.717923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5705.049443] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5715.898794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5716.065921] Lustre: 173019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5716.334742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5720.355237] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5728.677885] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5733.797665] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5740.223711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5740.519593] Lustre: lustre-OST0000: new disk, initializing [ 5740.521975] Lustre: Skipped 1 previous similar message [ 5740.525673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5740.532277] Lustre: Skipped 2 previous similar messages [ 5740.540478] Lustre: 174997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5746.203865] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5749.343600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5749.360346] Lustre: Skipped 1 previous similar message [ 5749.370510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5749.501651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5756.993472] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5765.096386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5765.100244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.116311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5765.137396] Lustre: Skipped 2 previous similar messages [ 5765.159068] Lustre: Skipped 8 previous similar messages [ 5771.420259] Lustre: server umount lustre-OST0000 complete [ 5777.890359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5777.913148] LustreError: Skipped 1 previous similar message [ 5790.176668] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5790.393483] Lustre: server umount lustre-MDT0000 complete [ 5790.689138] LustreError: 174217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.705852] LustreError: 174217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5794.221307] LustreError: 174996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788932017 with bad export cookie 7221316415359337844 [ 5794.234123] LustreError: 174996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5794.237312] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5794.511872] Lustre: server umount lustre-MDT0001 complete [ 5803.469976] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:33:44 (1788932024) [ 5820.020327] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 5830.726605] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5882.977712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5883.483090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5883.546879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5883.799626] Lustre: lustre-MDT0000: new disk, initializing [ 5883.947517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5883.961343] Lustre: Skipped 2 previous similar messages [ 5883.991280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5890.808313] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5906.085824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5906.272352] Lustre: 179325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5906.592455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5912.768442] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5923.209931] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5929.172518] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5936.885039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5937.209761] Lustre: 181305:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5943.896916] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5945.968477] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5946.109353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5954.749293] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5966.818421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5966.826745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.844458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5966.866638] Lustre: Skipped 3 previous similar messages [ 5966.885142] Lustre: Skipped 7 previous similar messages [ 5969.372161] Lustre: server umount lustre-OST0000 complete [ 5981.005733] Lustre: server umount lustre-MDT0000 complete [ 5982.182471] LustreError: 179338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.218968] LustreError: 179338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5985.363230] LustreError: 179318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788932208 with bad export cookie 7221316415359338929 [ 5985.367912] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5985.375657] LustreError: 179318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5985.959192] Lustre: server umount lustre-MDT0001 complete [ 5997.933531] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6000.767163] Key type lgssc unregistered [ 6001.101069] LNet: 183163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6001.115026] LNetError: 183163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6001.150840] LNet: Removed LNI 192.168.204.135@tcp [ 6002.270205] Key type .llcrypt unregistered [ 6002.280702] Key type ._llcrypt unregistered [ 6029.129196] Key type ._llcrypt registered [ 6029.137433] Key type .llcrypt registered [ 6029.262905] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6030.970264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6031.048151] alg: No test for adler32 (adler32-zlib) [ 6032.137693] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 6032.409071] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6034.127491] Key type lgssc registered [ 6035.828430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6045.410550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6045.461687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.216525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6051.525680] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6057.451857] LustreError: 184581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.495137] LustreError: 184581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6062.561975] LustreError: 184582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6062.682867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.185526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6068.650922] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6078.388409] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6085.059190] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6093.593381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6094.019839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.861150] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6109.201432] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6117.626847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6139.364825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6139.378729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6139.412584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6143.457415] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6143.466050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6143.476586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6143.496617] Lustre: Skipped 1 previous similar message [ 6144.978743] Lustre: server umount lustre-OST0000 complete [ 6153.700072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6153.741814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6153.752797] Lustre: Skipped 1 previous similar message [ 6156.783941] Lustre: server umount lustre-MDT0000 complete [ 6158.833206] LustreError: 184576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6158.868714] LustreError: 184576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6161.644248] LustreError: 184563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788932384 with bad export cookie 10966407070743547058 [ 6161.658200] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6161.670556] LustreError: 184563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6162.181743] Lustre: server umount lustre-MDT0001 complete [ 6175.001224] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6177.894537] Key type lgssc unregistered [ 6178.250731] LNet: 188149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6178.266126] LNetError: 188149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6178.281339] LNet: Removed LNI 192.168.204.135@tcp [ 6179.219409] Key type .llcrypt unregistered [ 6179.221905] Key type ._llcrypt unregistered [ 6205.782723] Key type ._llcrypt registered [ 6205.784108] Key type .llcrypt registered [ 6205.861351] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6207.101868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6207.225174] alg: No test for adler32 (adler32-zlib) [ 6208.286226] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 6208.446144] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6210.184235] Key type lgssc registered [ 6211.568906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6221.553582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6221.606565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6223.350312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.333944] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6233.569700] LustreError: 189566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.623380] LustreError: 189566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6238.688787] LustreError: 189567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6241.539116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.993068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6247.345055] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6255.179626] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6261.921702] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6270.207865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6270.810497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6279.709722] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6288.729984] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6293.368506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6299.899572] Lustre: lustre-OST0000: Not available for connect from 192.168.204.35@tcp (stopping) [ 6301.669811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6301.681717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6301.695589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6303.717094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6303.722737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.723256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6305.267736] Lustre: server umount lustre-OST0000 complete [ 6312.350778] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8b455244a000.ost_server_uuid 50 [ 6314.636437] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6329.321219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6329.344954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.364699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6329.372560] Lustre: Skipped 2 previous similar messages [ 6333.412106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6333.413537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6335.190745] Lustre: server umount lustre-MDT0000 complete [ 6338.528749] LustreError: 189566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6338.549949] LustreError: 189566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6339.147385] LustreError: 189546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788932562 with bad export cookie 14059325876488965652 [ 6339.161313] LustreError: 189546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6339.162420] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6339.571170] Lustre: server umount lustre-MDT0001 complete [ 6348.760534] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:42:50 (1788932570) [ 6355.849116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6356.410057] LustreError: 193483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6356.512831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6361.584775] LustreError: 193484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6361.982833] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6371.112977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6371.549062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6376.598650] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6384.849479] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6392.543214] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6399.800061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6400.353667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6401.391532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6408.873498] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6418.693574] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6426.079749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6426.364371] Lustre: lustre-OST0001: new disk, initializing [ 6426.372674] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6426.383927] Lustre: 196325:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6426.476228] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6434.084284] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6434.970171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6434.998384] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6435.189774] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6444.846989] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6450.664539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6450.674454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6450.700647] Lustre: Skipped 1 previous similar message [ 6450.712761] Lustre: Skipped 1 previous similar message [ 6454.128320] Lustre: server umount lustre-OST0000 complete [ 6455.781029] LustreError: 196333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6455.809275] LustreError: 196333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6463.277067] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6463.614138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6470.109671] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6470.490904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6476.279422] LustreError: 195306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6476.300329] LustreError: 195306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6486.500479] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6486.504577] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6486.516379] LustreError: Skipped 1 previous similar message [ 6486.524189] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6486.543700] Lustre: Skipped 1 previous similar message [ 6486.577588] Lustre: Skipped 3 previous similar messages [ 6490.904929] Lustre: server umount lustre-OST0001 complete [ 6502.053128] Lustre: server umount lustre-MDT0000 complete [ 6506.260394] LustreError: 193464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788932729 with bad export cookie 14059325876488966723 [ 6506.263448] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6506.267322] LustreError: 193464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6506.568699] Lustre: server umount lustre-MDT0001 complete [ 6534.025187] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:45:55 (1788932755) [ 6541.362264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6541.602568] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6541.658551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6541.968154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.502650] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6558.140951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6558.293637] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6558.378623] Lustre: 199768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6565.391511] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6574.953437] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6582.519023] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6592.680358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6593.046891] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6600.467810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6601.609792] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6610.080518] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6616.927371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6617.022560] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6617.398064] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6617.405664] Lustre: Skipped 2 previous similar messages [ 6624.299993] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6632.651764] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6636.026166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6636.034147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6636.041674] Lustre: Skipped 3 previous similar messages [ 6636.060096] Lustre: Skipped 3 previous similar messages [ 6642.033023] Lustre: server umount lustre-OST0000 complete [ 6643.918501] LustreError: 201943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6643.942489] LustreError: 201943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6661.601522] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6661.621892] LustreError: Skipped 1 previous similar message [ 6664.416965] Lustre: server umount lustre-OST0001 complete [ 6671.841628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6671.854867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6671.859163] Lustre: Skipped 3 previous similar messages [ 6674.329982] Lustre: server umount lustre-MDT0000 complete [ 6678.117460] LustreError: 202654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788932901 with bad export cookie 14059325876488968130 [ 6678.123565] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6678.136748] LustreError: 202654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6678.664549] Lustre: server umount lustre-MDT0001 complete [ 6702.652971] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:48:44 (1788932924) [ 6719.440370] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 6730.578137] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6778.949288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6779.349478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6779.435917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6779.597991] Lustre: lustre-MDT0000: new disk, initializing [ 6779.707518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6779.739323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6784.712143] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6796.888303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6797.102646] Lustre: 207864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6797.182880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6797.186567] Lustre: Skipped 1 previous similar message [ 6797.304034] Lustre: lustre-MDT0001: new disk, initializing [ 6797.516836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6797.523677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6803.855716] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6814.210941] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6820.245700] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6827.704213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6828.050713] Lustre: lustre-OST0000: new disk, initializing [ 6828.055032] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6828.060515] Lustre: 209836:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6835.115615] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6835.801807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6835.826650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6836.005818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6847.612868] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6853.305713] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6853.607244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6859.258528] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6859.750932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6866.917566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6866.939459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6866.995481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6867.016305] Lustre: Skipped 8 previous similar messages [ 6868.688174] Lustre: server umount lustre-OST0000 complete [ 6887.392103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6887.617394] Lustre: server umount lustre-MDT0000 complete [ 6891.943855] LustreError: 207858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933114 with bad export cookie 14059325876488969684 [ 6891.955426] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6891.961395] LustreError: 207858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6892.308139] Lustre: server umount lustre-MDT0001 complete [ 6901.450601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6902.037544] LustreError: 212055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6902.074727] LustreError: 212055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6907.048087] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6916.537792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6921.885166] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6929.993799] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6936.425185] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6944.170727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6951.955850] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6961.122232] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6966.180920] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6966.409232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6970.471922] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6970.757437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6973.410331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6973.417510] LustreError: Skipped 2 previous similar messages [ 6978.893732] Lustre: server umount lustre-OST0000 complete [ 6985.556682] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6990.120407] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7019.728383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7026.294338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7026.313560] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7026.603454] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7034.158889] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7038.954410] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 7041.713317] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7041.734396] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7047.141421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7047.149589] LustreError: Skipped 1 previous similar message [ 7047.152196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7047.167713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7047.174748] Lustre: Skipped 6 previous similar messages [ 7047.191964] Lustre: Skipped 11 previous similar messages [ 7049.633832] Lustre: server umount lustre-OST0000 complete [ 7060.436506] Lustre: server umount lustre-MDT0000 complete [ 7062.497797] LustreError: 212050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7062.509638] LustreError: 212050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7064.159402] LustreError: 212035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933287 with bad export cookie 14059325876488970573 [ 7064.172709] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7064.572530] Lustre: server umount lustre-MDT0001 complete [ 7074.219169] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:54:55 (1788933295) [ 7082.171559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7082.867883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7082.874765] Lustre: Skipped 6 previous similar messages [ 7087.927263] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7097.133895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7101.498989] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7109.512468] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7116.246815] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7124.011541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7131.157167] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7139.193926] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7144.677847] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7144.989928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7150.324130] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7150.636955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7158.202324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7158.453080] Lustre: lustre-OST0001: new disk, initializing [ 7158.459474] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7158.469525] Lustre: 221391:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7164.532095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7164.547723] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7164.622561] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7165.167885] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7176.036524] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7182.324309] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7182.544460] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7187.733161] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7188.009139] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7204.319215] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7204.562089] Lustre: server umount lustre-OST0001 complete [ 7212.368955] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7212.607646] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7217.754696] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7217.966304] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7248.087382] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7253.995585] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7254.002835] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7254.004668] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7256.296527] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7266.466097] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7268.597829] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7274.464151] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7274.477031] LustreError: Skipped 1 previous similar message [ 7277.239848] Lustre: server umount lustre-OST0001 complete [ 7302.543726] Lustre: server umount lustre-OST0000 complete [ 7307.246828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7307.259250] Lustre: Skipped 8 previous similar messages [ 7307.267936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7307.273074] Lustre: Skipped 16 previous similar messages [ 7313.532445] Lustre: server umount lustre-MDT0000 complete [ 7317.764566] LustreError: 218002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933540 with bad export cookie 14059325876488971539 [ 7317.779282] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7317.797353] LustreError: 218002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7332.319510] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7332.670622] Lustre: server umount lustre-MDT0001 complete [ 7362.891492] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:59:43 (1788933583) [ 7373.010866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7373.627312] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7373.705417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7380.273791] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7393.516868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7393.792565] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7393.878471] Lustre: 226642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7401.263392] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7412.212766] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7418.820824] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7428.025431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7428.423269] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7436.964439] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7446.921830] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7457.461353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7457.614252] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7464.310194] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7472.449656] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7474.618494] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7480.370213] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7480.609463] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7486.696610] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7487.032464] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7493.526610] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b4542bbf800.ost_server_uuid 50 [ 7495.354251] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b4542bbf800.ost_server_uuid in IDLE state after 0 sec [ 7496.365291] Lustre: Permanently deactivating lustre-OST0001 [ 7496.389537] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7499.952244] Lustre: Permanently reactivating lustre-OST0001 [ 7510.238935] Lustre: server umount lustre-OST0001 complete [ 7511.530218] LustreError: 228495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7511.542627] LustreError: 228495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7520.898419] Lustre: server umount lustre-OST0000 complete [ 7539.169180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7539.313034] Lustre: server umount lustre-MDT0000 complete [ 7542.988463] LustreError: 226634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933766 with bad export cookie 14059325876488972827 [ 7542.990354] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7543.003584] LustreError: 226634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7543.375802] Lustre: server umount lustre-MDT0001 complete [ 7569.386205] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:03:10 (1788933790) [ 7577.383476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7577.722301] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7577.773131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7577.783704] Lustre: Skipped 5 previous similar messages [ 7583.576764] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7593.688238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7594.039627] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7594.094397] Lustre: 233355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7602.713612] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7613.726866] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7619.991994] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7628.427804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7628.745521] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7628.793264] Lustre: Permanently deactivating lustre-OST0000 [ 7628.812753] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7629.085730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7629.098830] Lustre: Skipped 10 previous similar messages [ 7636.201902] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7638.547578] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7644.025984] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7652.135580] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7652.246389] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7658.295355] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7660.120765] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7669.664921] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7671.675926] Lustre: Permanently reactivating lustre-OST0000 [ 7671.707765] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7671.716542] Lustre: Skipped 2 previous similar messages [ 7681.098071] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7681.098523] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 43 seconds. I think it's dead, and I am evicting it. exp 00000000048e9613, cur 1788933904 deadline 1788933891 last 1788933861 [ 7681.104153] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7681.112644] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7681.149076] Lustre: Skipped 1 previous similar message [ 7681.195509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7689.190329] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7689.201210] LustreError: Skipped 7 previous similar messages [ 7693.593714] Lustre: server umount lustre-OST0001 complete [ 7712.226521] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7712.535311] Lustre: server umount lustre-OST0000 complete [ 7723.920887] Lustre: server umount lustre-MDT0000 complete [ 7727.919297] LustreError: 233348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933950 with bad export cookie 14059325876488974479 [ 7727.923587] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7727.940300] LustreError: 233348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7728.387387] Lustre: server umount lustre-MDT0001 complete [ 7745.967531] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 7749.404372] Key type lgssc unregistered [ 7749.793497] LNet: 238246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7749.805845] LNetError: 238246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7749.830625] LNet: Removed LNI 192.168.204.135@tcp [ 7751.011348] Key type .llcrypt unregistered [ 7751.013698] Key type ._llcrypt unregistered [ 7775.810891] Key type ._llcrypt registered [ 7775.813335] Key type .llcrypt registered [ 7775.965550] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 7777.101182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7777.170169] alg: No test for adler32 (adler32-zlib) [ 7778.215862] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 7778.446781] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 7780.159171] Key type lgssc registered [ 7781.322779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7792.325812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7792.356581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7793.831932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7798.909904] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7803.384635] LustreError: 239732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7803.427449] LustreError: 239732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7808.487214] LustreError: 239733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7809.569367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7809.844428] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7809.916429] Lustre: 239722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7809.943250] Lustre: Permanently deactivating lustre-MDT0001 [ 7809.981582] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7810.523385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7816.247619] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7817.280471] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7824.609189] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7830.869258] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7838.901414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7839.285945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7846.235728] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7854.895766] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7855.664720] LustreError: 241573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7860.708558] LustreError: 241575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7863.801566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7864.252806] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7864.495976] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7869.934643] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7869.946852] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7871.098611] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7879.024444] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7883.341423] Lustre: Permanently reactivating lustre-MDT0001 [ 7883.357948] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7883.362206] Lustre: Skipped 2 previous similar messages [ 7886.191722] Lustre: lustre-MDT0001: Client 425f4bcb-5603-4b9c-be05-b8c07c221fa0 (at 192.168.204.35@tcp) reconnecting [ 7891.014359] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7891.027871] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7891.038688] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7891.053625] LustreError: 243445:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7891.061061] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7891.067401] LustreError: 243445:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7891.069352] LustreError: 243445:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.135@tcp: cfg command failed: rc = -108 [ 7891.114654] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7891.114654] [ 7891.123401] LustreError: 239724:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7892.780504] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7894.465344] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7900.585285] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7900.843916] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7905.102976] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7905.350798] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7908.619399] Lustre: Permanently deactivating lustre-MDT0001 [ 7908.633004] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7908.636992] Lustre: Skipped 2 previous similar messages [ 7917.156682] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7923.169060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7923.179179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7923.207421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7925.098279] Lustre: server umount lustre-MDT0000 complete [ 7927.269909] LustreError: 239733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7927.275761] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7927.285782] LustreError: 239733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7927.314415] Lustre: Skipped 1 previous similar message [ 7929.697571] LustreError: 240472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934152 with bad export cookie 15966410103707938489 [ 7929.709039] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7930.100727] Lustre: server umount lustre-MDT0001 complete [ 7936.490756] Lustre: server umount lustre-OST0000 complete [ 7949.660482] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:09:30 (1788934170) [ 7962.741833] Lustre: server umount lustre-OST0001 complete [ 7971.288193] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 7983.029883] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 8021.682697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8021.929629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8021.945276] Lustre: Skipped 2 previous similar messages [ 8021.973297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8022.098152] Lustre: lustre-MDT0000: new disk, initializing [ 8022.154883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8022.186379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8026.418565] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8037.014371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8037.152322] Lustre: 247993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8037.198448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8037.203123] Lustre: Skipped 1 previous similar message [ 8037.262281] Lustre: lustre-MDT0001: new disk, initializing [ 8037.354333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8037.410336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8037.423434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8041.281950] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8050.408511] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8055.919736] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8063.584082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8063.869633] Lustre: lustre-OST0000: new disk, initializing [ 8063.872826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8063.878702] Lustre: 249959:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8063.965050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8070.203614] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8071.812694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8071.829146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8072.007762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8080.356905] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8082.523608] LustreError: 247999:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8091.510948] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8091.730570] Lustre: lustre-OST0001: new disk, initializing [ 8091.734404] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8091.743790] Lustre: 251074:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8092.549378] LustreError: 247999:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8092.597091] LustreError: 247997:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8099.030214] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8099.612721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8099.621435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8099.777148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8102.642486] LustreError: 247997:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8111.884205] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8113.823604] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8120.294218] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8120.295738] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8120.317599] LustreError: Skipped 1 previous similar message [ 8120.320320] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8120.347166] Lustre: Skipped 2 previous similar messages [ 8120.365528] Lustre: Skipped 1 previous similar message [ 8121.931756] Lustre: server umount lustre-OST0001 complete [ 8125.420387] LustreError: 249941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8125.440112] LustreError: 249941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8140.773729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8140.808276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8140.816463] Lustre: Skipped 1 previous similar message [ 8145.894274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8145.906181] LustreError: 249941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8145.906192] LustreError: 249941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8145.940109] Lustre: Skipped 1 previous similar message [ 8146.715888] Lustre: server umount lustre-OST0000 complete [ 8155.108381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8155.119233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8155.129406] Lustre: Skipped 1 previous similar message [ 8155.134981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8157.710548] Lustre: server umount lustre-MDT0000 complete [ 8162.373220] LustreError: 247985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934385 with bad export cookie 15966410103707941086 [ 8162.396222] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8162.399237] LustreError: 247985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8169.179611] Lustre: server umount lustre-MDT0001 complete [ 8182.302230] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 8185.178187] Key type lgssc unregistered [ 8185.561703] LNet: 253112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8185.566355] LNetError: 253112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8185.583773] LNet: Removed LNI 192.168.204.135@tcp [ 8186.624502] Key type .llcrypt unregistered [ 8186.627601] Key type ._llcrypt unregistered [ 8227.723458] Key type ._llcrypt registered [ 8227.726675] Key type .llcrypt registered [ 8227.861643] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 8229.053432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8229.177721] alg: No test for adler32 (adler32-zlib) [ 8230.229072] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 8230.532500] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 8232.223177] Key type lgssc registered [ 8233.393175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8240.769771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8240.803851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8242.106255] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8242.143202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8242.490171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8247.902340] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8258.799477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8258.883129] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8258.920316] Lustre: 255320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8259.180759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8264.002671] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8273.082938] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8280.047732] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8288.025058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8288.187422] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8288.362595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8295.662659] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8296.632247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8303.975707] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8358.371082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8358.374751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8358.387989] LustreError: Skipped 1 previous similar message [ 8358.388741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8358.427550] Lustre: Skipped 1 previous similar message [ 8363.496573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8363.509868] Lustre: Skipped 1 previous similar message [ 8363.784090] Lustre: server umount lustre-OST0000 complete [ 8368.747231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8383.935539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8384.261354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8388.353173] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8388.373045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8388.374626] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8388.394885] Lustre: Skipped 1 previous similar message [ 8392.122797] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8401.522888] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8444.899647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8444.903760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8444.930199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8444.939711] Lustre: Skipped 2 previous similar messages [ 8447.965699] Lustre: server umount lustre-OST0000 complete [ 8453.600177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8453.610513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8453.622704] Lustre: Skipped 1 previous similar message [ 8453.631814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8458.718509] Lustre: server umount lustre-MDT0000 complete [ 8462.901178] LustreError: 255312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934685 with bad export cookie 10375462269843298359 [ 8462.904152] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8462.920597] LustreError: 255312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8463.382553] Lustre: server umount lustre-MDT0001 complete [ 8474.463370] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 8477.648286] Key type lgssc unregistered [ 8478.058679] LNet: 260748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8478.077775] LNetError: 260748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8479.142513] LNet: Removed LNI 192.168.204.135@tcp [ 8480.074168] Key type .llcrypt unregistered [ 8480.080310] Key type ._llcrypt unregistered [ 8504.405924] Key type ._llcrypt registered [ 8504.411837] Key type .llcrypt registered [ 8504.525841] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 8505.598873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8505.638608] alg: No test for adler32 (adler32-zlib) [ 8506.744965] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 8507.003634] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 8508.719392] Key type lgssc registered [ 8510.003851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8518.451616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8518.464919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8520.177958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8524.676193] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8530.417593] LustreError: 262182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8530.443446] LustreError: 262182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8534.479165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8534.987945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8540.191108] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8548.941843] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8554.947721] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8562.459657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8562.958986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8565.070773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8570.777599] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8578.971373] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8610.275234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8610.277888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8610.305915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8613.754524] Lustre: server umount lustre-OST0000 complete [ 8620.518474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8620.534760] Lustre: Skipped 1 previous similar message [ 8620.544726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8620.556564] Lustre: Skipped 2 previous similar messages [ 8625.648726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8630.751994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8630.756239] Lustre: Skipped 1 previous similar message [ 8632.287153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8632.581955] Lustre: server umount lustre-MDT0000 complete [ 8636.897433] LustreError: 262179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8636.905902] LustreError: 262179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8637.147629] LustreError: 264025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934860 with bad export cookie 1524875430908838008 [ 8637.148644] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8637.167229] LustreError: 264025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8637.645197] Lustre: server umount lustre-MDT0001 complete [ 8651.205331] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 8655.267781] Key type lgssc unregistered [ 8655.790551] LNet: 266495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8655.798720] LNetError: 266495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8655.819814] LNet: Removed LNI 192.168.204.135@tcp [ 8656.823374] Key type .llcrypt unregistered [ 8656.828346] Key type ._llcrypt unregistered [ 8680.551479] Key type ._llcrypt registered [ 8680.554588] Key type .llcrypt registered [ 8680.710567] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 8682.258457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8682.367069] alg: No test for adler32 (adler32-zlib) [ 8683.503367] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 8683.889133] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 8685.639877] Key type lgssc registered [ 8687.193782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8696.470763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8696.489890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8698.055813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8704.270956] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8708.581345] LustreError: 267719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8708.616084] LustreError: 267719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8713.701066] LustreError: 267718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8715.557227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8716.271699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8722.115484] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8732.496152] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8740.076611] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8748.140142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.499484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8756.121117] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8766.771176] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8769.020264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8787.424926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8787.445404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8787.482978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8789.474543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8789.478067] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8789.494311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8791.756477] Lustre: server umount lustre-OST0000 complete [ 8798.177146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8798.193777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8798.230869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8798.242864] Lustre: Skipped 1 previous similar message [ 8803.449042] Lustre: server umount lustre-MDT0000 complete [ 8804.835938] LustreError: 268447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8804.880038] LustreError: 268447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8809.198066] LustreError: 267698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935032 with bad export cookie 16938712653262631496 [ 8809.226145] LustreError: 267698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8809.246400] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8809.627178] Lustre: server umount lustre-MDT0001 complete [ 8822.614584] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 8826.458339] Key type lgssc unregistered [ 8826.844985] LNet: 271403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8826.858254] LNetError: 271403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8827.879902] LNet: Removed LNI 192.168.204.135@tcp [ 8829.089756] Key type .llcrypt unregistered [ 8829.094307] Key type ._llcrypt unregistered [ 8858.967905] Key type ._llcrypt registered [ 8858.970716] Key type .llcrypt registered [ 8859.198969] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 8860.417975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8860.503991] alg: No test for adler32 (adler32-zlib) [ 8861.770730] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 8862.155028] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 8863.967310] Key type lgssc registered [ 8865.281698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8874.024752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8874.071939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8875.700393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8880.762945] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8885.227769] LustreError: 272822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8885.256644] LustreError: 272822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8890.342874] LustreError: 272823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8890.581508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8891.244803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8896.450820] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8905.235199] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8911.567887] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8919.919862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8920.462027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8921.530083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8927.548684] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 8936.613238] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8962.529633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8962.533527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8962.555686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8967.656288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8967.662814] Lustre: Skipped 2 previous similar messages [ 8968.348757] Lustre: server umount lustre-OST0000 complete [ 8973.284879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8973.291810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8973.301995] Lustre: Skipped 1 previous similar message [ 8973.317114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8977.889117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8977.890389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8979.421409] Lustre: server umount lustre-MDT0000 complete [ 8983.008080] LustreError: 273550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8983.040621] LustreError: 273550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8983.385571] LustreError: 272804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935206 with bad export cookie 8445019091003824953 [ 8983.400980] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8983.406543] LustreError: 272804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8983.826539] Lustre: server umount lustre-MDT0001 complete [ 8997.637064] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 9000.804917] Key type lgssc unregistered [ 9001.118871] LNet: 277132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9001.137535] LNetError: 277132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9001.164939] LNet: Removed LNI 192.168.204.135@tcp [ 9002.394311] Key type .llcrypt unregistered [ 9002.395787] Key type ._llcrypt unregistered [ 9024.975261] Key type ._llcrypt registered [ 9024.977601] Key type .llcrypt registered [ 9025.112270] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 9026.825968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9026.856908] alg: No test for adler32 (adler32-zlib) [ 9028.061515] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 9028.469873] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 9030.351868] Key type lgssc registered [ 9032.006525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9041.555086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9041.596387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9043.433241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9050.157413] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9053.666865] LustreError: 278368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9053.685992] LustreError: 278368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9058.790746] LustreError: 278370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9061.355513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9061.861980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9067.592345] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9076.611408] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9083.980759] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9091.841148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9092.322230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9099.889320] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9108.540989] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9113.918129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9126.877262] Lustre: server umount lustre-OST0000 complete [ 9133.541265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9133.546148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9133.573915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9134.564633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9134.568372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9139.680883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9139.698094] Lustre: Skipped 2 previous similar messages [ 9143.775435] Lustre: 277665:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788935350/real 1788935350] req@00000000467dcc99 x1875834569784064/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788935366 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9143.799724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9143.823601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9143.833219] Lustre: Skipped 1 previous similar message [ 9145.823227] Lustre: 277664:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788935352/real 1788935352] req@0000000028c5cf2b x1875834569785088/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788935368 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9146.336920] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9146.771906] Lustre: server umount lustre-MDT0000 complete [ 9148.902241] LustreError: 278370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9148.924006] LustreError: 278370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9150.943145] Lustre: 277667:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788935357/real 1788935357] req@0000000004d874cb x1875834569787008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788935373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9151.879603] LustreError: 278334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935374 with bad export cookie 6885460900482778817 [ 9151.901231] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9152.369091] Lustre: server umount lustre-MDT0001 complete [ 9165.606442] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 9168.951776] Key type lgssc unregistered [ 9169.273169] LNet: 282077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9169.278920] LNetError: 282077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9169.296834] LNet: Removed LNI 192.168.204.135@tcp [ 9170.495348] Key type .llcrypt unregistered [ 9170.497247] Key type ._llcrypt unregistered [ 9182.762067] Key type ._llcrypt registered [ 9182.770741] Key type .llcrypt registered [ 9182.907254] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9203.427974] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 9221.753649] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 9223.439217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9223.486475] alg: No test for adler32 (adler32-zlib) [ 9224.684938] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 9224.951278] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 9226.616574] Key type lgssc registered [ 9228.009783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9266.876842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9266.924575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9268.258840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9268.288529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9268.351201] Lustre: lustre-MDT0000: new disk, initializing [ 9268.416967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9268.430696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9273.367430] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9286.574883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9286.782247] Lustre: 285635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9286.862686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9286.866550] Lustre: Skipped 1 previous similar message [ 9287.021412] Lustre: lustre-MDT0001: new disk, initializing [ 9287.159199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9287.208535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9287.228029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9293.513600] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9305.448976] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9311.673062] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9319.675229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9320.149287] Lustre: lustre-OST0000: new disk, initializing [ 9320.154431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9320.169831] Lustre: 287603:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9320.264650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9327.527416] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9329.254230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9329.264488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9329.423573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9338.679584] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9344.187443] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9344.474652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9349.970882] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9350.360137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9355.239265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9355.242976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9355.248832] LustreError: Skipped 1 previous similar message [ 9355.255128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9355.272642] Lustre: Skipped 1 previous similar message [ 9358.643830] Lustre: server umount lustre-OST0000 complete [ 9363.937753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9363.957642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9363.984084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9363.998419] Lustre: Skipped 1 previous similar message [ 9365.473744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9365.474071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9368.976910] Lustre: server umount lustre-MDT0000 complete [ 9374.624397] LustreError: 287613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935597 with bad export cookie 7800454312933004167 [ 9374.632888] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9374.643654] LustreError: 287613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9375.287372] Lustre: server umount lustre-MDT0001 complete [ 9387.038773] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:33:28 (1788935608) [ 9394.291436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9394.760866] LustreError: 290015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9394.784615] LustreError: 290015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9394.878688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9400.176674] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9400.290395] LustreError: 290016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9404.394888] LustreError: 290015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9409.504810] LustreError: 290016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9411.123971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9411.643615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9416.858908] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9426.281239] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9433.936497] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9444.089096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9444.551825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9452.945494] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9463.151902] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9477.606142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9477.613852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9477.640186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9477.651064] Lustre: Skipped 1 previous similar message [ 9480.688521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9482.786233] Lustre: server umount lustre-OST0000 complete [ 9488.363031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9488.378139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9488.395722] Lustre: Skipped 1 previous similar message [ 9488.409846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9488.421859] Lustre: Skipped 2 previous similar messages [ 9493.648201] Lustre: server umount lustre-MDT0000 complete [ 9496.033707] LustreError: 290011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9496.052665] LustreError: 290011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9498.456517] LustreError: 289995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935721 with bad export cookie 7800454312933005049 [ 9498.469285] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9498.953670] Lustre: server umount lustre-MDT0001 complete [ 9511.106949] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 9514.312161] Key type lgssc unregistered [ 9514.644774] LNet: 293576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9514.659155] LNetError: 293576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9514.685063] LNet: Removed LNI 192.168.204.135@tcp [ 9515.698218] Key type .llcrypt unregistered [ 9515.700869] Key type ._llcrypt unregistered [ 9548.046819] Key type ._llcrypt registered [ 9548.048895] Key type .llcrypt registered [ 9548.155444] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 9567.397814] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 9569.434336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9569.473601] alg: No test for adler32 (adler32-zlib) [ 9570.750638] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 9571.094955] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 9572.767198] Key type lgssc registered [ 9574.506337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9633.826589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9633.863676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9635.145097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9635.165330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9635.265153] Lustre: lustre-MDT0000: new disk, initializing [ 9635.369527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9635.391346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9640.652309] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9652.783057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9652.926260] Lustre: 297414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9652.998982] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9653.004308] Lustre: Skipped 1 previous similar message [ 9653.092511] Lustre: lustre-MDT0001: new disk, initializing [ 9653.176230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9653.215219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9653.227609] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9658.053872] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9668.016290] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9674.737757] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9682.261177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9682.654172] Lustre: lustre-OST2710: new disk, initializing [ 9682.659404] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9682.668593] Lustre: 299384:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9682.767990] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9689.820696] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9690.261102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9690.285866] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9690.438122] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9700.214939] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9707.431796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9707.659323] Lustre: lustre-OST03e8: new disk, initializing [ 9707.666938] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9707.674828] Lustre: 300446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9707.807372] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9714.814288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9714.841304] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9715.015160] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9715.661873] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9724.337785] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9732.868953] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9733.136114] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9738.907956] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9739.322356] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9745.661537] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9745.957574] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9751.689106] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9751.958770] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9759.546788] Lustre: server umount lustre-MDT0000 complete [ 9760.736360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9760.748279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9760.760952] LustreError: 297425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9760.779483] LustreError: 297425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9761.262801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9761.277539] LustreError: 297419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9761.306402] LustreError: 297419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9764.104956] LustreError: 297405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935987 with bad export cookie 11251564720144624989 [ 9764.117558] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9764.128301] LustreError: 297405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9764.743749] Lustre: server umount lustre-MDT0001 complete [ 9779.289341] Lustre: server umount lustre-OST2710 complete [ 9782.176623] Lustre: 295212:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788935989/real 1788935989] req@00000000b2b07f70 x1875835139201664/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788936005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9782.210808] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9782.240837] Lustre: Skipped 2 previous similar messages [ 9783.288282] Lustre: server umount lustre-OST03e8 complete [ 9802.850785] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 9812.397902] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 9853.524312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9853.867799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9853.920239] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9854.067406] Lustre: lustre-MDT0000: new disk, initializing [ 9854.144846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9854.167455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9859.580407] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9871.070896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9871.309918] Lustre: 305696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9871.423990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9871.437798] Lustre: Skipped 1 previous similar message [ 9871.559761] Lustre: lustre-MDT0001: new disk, initializing [ 9871.700919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9871.779132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9871.788749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9876.949330] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9888.273466] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9895.485907] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9905.066889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9905.499715] Lustre: lustre-OST0000: new disk, initializing [ 9905.506347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9905.515553] Lustre: 307666:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9905.657737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9912.906233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9912.922452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9913.061877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9913.222359] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 9925.731344] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9930.970587] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9931.324483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9936.131919] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9936.394305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9938.914517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9938.915093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9938.930785] LustreError: Skipped 1 previous similar message [ 9938.936180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9938.963871] Lustre: Skipped 1 previous similar message [ 9944.036106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9944.050174] Lustre: Skipped 2 previous similar messages [ 9944.619290] Lustre: server umount lustre-OST0000 complete [ 9948.640871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9948.645994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9948.657498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9954.273522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9954.278557] Lustre: Skipped 3 previous similar messages [ 9955.098649] Lustre: server umount lustre-MDT0000 complete [ 9959.062294] LustreError: 305690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936182 with bad export cookie 11251564720144626669 [ 9959.065844] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9959.069520] LustreError: 305690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9959.433960] Lustre: server umount lustre-MDT0001 complete [ 9968.867234] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:43:09 (1788936189) [ 9970.664876] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9972.537496] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:43:13 (1788936193) [ 9976.035365] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [10005.659270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10006.035417] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10006.049681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10006.169769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10010.998917] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10020.907593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10021.056297] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10021.122585] Lustre: 311527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10026.149697] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10047.994904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10048.280799] Lustre: Denying initial registration attempt from nid 192.168.204.135@tcp, specified as failover [10048.290499] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10048.299943] LustreError: 312979:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10048.306288] LustreError: 312979:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10048.311720] LustreError: 312979:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10048.395879] Lustre: server umount lustre-OST0000 complete [10048.404264] LustreError: 312979:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10057.196578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10057.210903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10057.239136] Lustre: Skipped 1 previous similar message [10057.249689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10062.624252] Lustre: server umount lustre-MDT0000 complete [10062.817801] LustreError: 311539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10062.847269] LustreError: 311539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10067.750508] LustreError: 312268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936290 with bad export cookie 11251564720144627544 [10067.754971] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10067.776567] LustreError: 312268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10068.281987] Lustre: server umount lustre-MDT0001 complete [10080.969466] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [10084.225428] Key type lgssc unregistered [10084.602601] LNet: 314103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10084.624495] LNetError: 314103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10084.649591] LNet: Removed LNI 192.168.204.135@tcp [10085.744354] Key type .llcrypt unregistered [10085.747190] Key type ._llcrypt unregistered [10100.074343] Key type ._llcrypt registered [10100.076432] Key type .llcrypt registered [10100.170725] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [10101.620921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10101.678400] alg: No test for adler32 (adler32-zlib) [10103.183981] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [10103.480089] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [10105.215417] Key type lgssc registered [10107.039322] Lustre: Echo OBD driver; http://www.lustre.org/ [10154.587645] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [10163.262881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10163.277269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10164.530792] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10164.571518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10164.752364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10169.949588] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10179.532184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10179.700599] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10179.785438] Lustre: 316877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10180.215108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10184.997952] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10193.880257] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10201.042909] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10222.787391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10223.208706] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10223.411726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10232.786193] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10243.082950] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10248.172533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10248.187219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10248.221969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10252.042538] Lustre: server umount lustre-OST0000 complete [10258.924385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10258.936986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10258.964582] Lustre: Skipped 2 previous similar messages [10258.971865] Lustre: Skipped 1 previous similar message [10264.035322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10264.046185] Lustre: Skipped 1 previous similar message [10268.137228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10268.147755] Lustre: Skipped 1 previous similar message [10272.229919] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10272.771521] Lustre: server umount lustre-MDT0000 complete [10277.629402] LustreError: 316870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936500 with bad export cookie 9080370714529842439 [10277.651241] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10277.657421] LustreError: 316870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10278.159367] Lustre: server umount lustre-MDT0001 complete [10296.786958] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [10300.973190] Key type lgssc unregistered [10301.566608] LNet: 320481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10301.577802] LNetError: 320481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10301.604200] LNet: Removed LNI 192.168.204.135@tcp [10303.073906] Key type .llcrypt unregistered [10303.076740] Key type ._llcrypt unregistered [10337.655478] Key type ._llcrypt registered [10337.659807] Key type .llcrypt registered [10337.776415] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [10339.026296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10339.055109] alg: No test for adler32 (adler32-zlib) [10340.140833] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [10340.599140] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [10342.423261] Key type lgssc registered [10343.950379] Lustre: Echo OBD driver; http://www.lustre.org/ [10353.292100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10353.350265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10354.975138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10361.228267] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10364.389374] LustreError: 321900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10364.417567] LustreError: 321900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10369.517305] LustreError: 321899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10371.570797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10372.206770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10377.334608] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10387.175967] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10393.929378] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10403.587574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10404.207467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10410.994211] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10420.378689] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10436.859113] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) [10437.900403] Lustre: server umount lustre-MDT0000 complete [10438.627217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10438.637583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10438.659402] LustreError: 321899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10440.184044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10440.205507] Lustre: Skipped 1 previous similar message [10441.941124] LustreError: 321896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10441.981087] LustreError: 321896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10443.724232] LustreError: 321879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936666 with bad export cookie 5849267081291784881 [10443.742564] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10445.289064] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10445.312191] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10447.055142] LustreError: 324207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10447.080285] LustreError: 324207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10447.110989] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.35@tcp (stopping) [10452.189219] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.35@tcp (stopping) [10452.207360] Lustre: Skipped 1 previous similar message [10457.294674] LustreError: 324019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10457.294928] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.35@tcp (stopping) [10457.327288] LustreError: 324019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10457.356484] Lustre: Skipped 1 previous similar message [10458.079452] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10458.741456] Lustre: server umount lustre-MDT0001 complete [10467.144466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10477.425544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10488.293672] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x512cc56b70c87fc9 [10488.312304] Lustre: MGC192.168.204.135@tcp: Connection restored to 0@lo (at 0@lo) [10488.654463] LustreError: 323732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10488.727960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10488.785303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10493.979949] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10503.748714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10504.158169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10508.272754] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10509.532417] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10517.736374] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10519.341462] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10525.777563] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10527.634710] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10533.861097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10533.863974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10533.876941] Lustre: Skipped 1 previous similar message [10540.082331] Lustre: server umount lustre-OST0000 complete [10545.121245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10545.135858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10550.598578] Lustre: server umount lustre-MDT0000 complete [10554.348274] LustreError: 325523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10554.374181] LustreError: 325523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10555.245452] LustreError: 321881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936778 with bad export cookie 5849267081291792329 [10555.255766] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10555.259146] LustreError: 321881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10555.563899] Lustre: server umount lustre-MDT0001 complete [10571.512897] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [10575.194450] Key type lgssc unregistered [10575.459699] LNet: 328197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10575.487177] LNetError: 328197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10575.508294] LNet: Removed LNI 192.168.204.135@tcp [10577.116644] Key type .llcrypt unregistered [10577.121108] Key type ._llcrypt unregistered [10611.643265] Key type ._llcrypt registered [10611.653898] Key type .llcrypt registered [10611.783175] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [10613.702065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10614.075148] alg: No test for adler32 (adler32-zlib) [10615.483888] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [10615.983107] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [10617.727196] Key type lgssc registered [10620.070901] Lustre: Echo OBD driver; http://www.lustre.org/ [10631.405429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10631.449783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10633.163944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10638.943315] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10643.427783] LustreError: 329617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10643.448777] LustreError: 329617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10648.551877] LustreError: 329618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10650.037690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10650.542673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10656.416755] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10665.027626] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10674.079648] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10681.312559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10681.321405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10681.341702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10683.365668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.383867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10683.416320] Lustre: Skipped 1 previous similar message [10686.366589] Lustre: server umount lustre-MDT0000 complete [10688.492663] LustreError: 329617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10691.276836] LustreError: 329599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936914 with bad export cookie 16379529940552731582 [10691.279482] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10691.296206] LustreError: 329599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10691.991523] Lustre: server umount lustre-MDT0001 complete [10711.582052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10712.133373] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10712.205570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10712.473260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10717.991727] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10730.123433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10730.385256] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10730.489828] Lustre: 332205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10731.063851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10737.545350] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10748.039325] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10755.007144] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10766.526320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10767.005827] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10767.017278] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10767.040989] LustreError: 334094:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10767.057413] LustreError: 334094:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10767.067939] LustreError: 334094:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10767.176508] Lustre: server umount lustre-OST0000 complete [10767.180887] LustreError: 334094:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10781.500074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10781.891836] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10782.203487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10789.642237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10792.376968] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10805.194807] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10817.012416] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10817.136224] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10817.208803] Lustre: lustre-OST0001: new disk, initializing [10817.214725] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10817.220780] Lustre: 335638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10817.333195] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10823.768134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10823.775145] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10823.846565] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10825.922575] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [10836.886249] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10844.641838] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10844.644565] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10844.656614] LustreError: Skipped 1 previous similar message [10844.661093] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10844.733127] Lustre: Skipped 1 previous similar message [10846.674188] Lustre: server umount lustre-OST0001 complete [10849.781875] LustreError: 334564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10849.800473] LustreError: 334564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10854.883961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10854.885260] LustreError: 334563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10854.885946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10854.885951] Lustre: Skipped 1 previous similar message [10854.901422] Lustre: Skipped 1 previous similar message [10854.920355] LustreError: 334563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10857.820735] Lustre: server umount lustre-OST0000 complete [10864.035698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10864.056044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10864.068602] Lustre: Skipped 2 previous similar messages [10870.051008] Lustre: server umount lustre-MDT0000 complete [10874.337100] LustreError: 333215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.337100] LustreError: 334095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10875.118163] LustreError: 332200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937098 with bad export cookie 16379529940552732135 [10875.124186] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10875.137314] LustreError: 332200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10875.675556] Lustre: server umount lustre-MDT0001 complete [10893.425456] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [10897.645677] Key type lgssc unregistered [10897.961615] LNet: 337605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10897.979502] LNetError: 337605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10898.008445] LNet: Removed LNI 192.168.204.135@tcp [10899.075145] Key type .llcrypt unregistered [10899.079047] Key type ._llcrypt unregistered [10994.369930] Key type ._llcrypt registered [10994.371766] Key type .llcrypt registered [10994.571430] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [11012.253655] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [11013.744254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11013.776297] alg: No test for adler32 (adler32-zlib) [11015.210502] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [11015.578922] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [11017.391210] Key type lgssc registered [11019.219905] Lustre: Echo OBD driver; http://www.lustre.org/ [11070.358699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11070.446530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11071.915499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11071.976396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11072.061842] Lustre: lustre-MDT0000: new disk, initializing [11072.175320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11072.224492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11079.464952] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11091.988515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11092.083019] Lustre: 342559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11092.166819] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11092.172861] Lustre: Skipped 1 previous similar message [11092.287822] Lustre: lustre-MDT0001: new disk, initializing [11092.416727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11092.515044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11092.526918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11096.912900] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11109.808024] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11118.405565] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11126.885881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11127.488127] Lustre: lustre-OST0000: new disk, initializing [11127.502122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11127.534545] Lustre: 344529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11127.733163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11133.156395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11133.174051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11133.529903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11137.899493] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11150.828666] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11158.297035] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11158.728753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11166.851242] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11167.210532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11174.376297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11174.386458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11174.395248] LustreError: Skipped 1 previous similar message [11174.419683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11174.430908] Lustre: Skipped 1 previous similar message [11179.501059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11184.612468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11184.621037] Lustre: Skipped 2 previous similar messages [11185.119180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11185.271624] Lustre: server umount lustre-OST0000 complete [11194.852177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11194.854237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11194.880669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11194.890693] Lustre: Skipped 2 previous similar messages [11194.902122] Lustre: Skipped 1 previous similar message [11197.862698] Lustre: server umount lustre-MDT0000 complete [11202.935234] LustreError: 342550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937425 with bad export cookie 10826925217412340323 [11202.936941] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11202.949881] LustreError: 342550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11203.623599] Lustre: server umount lustre-MDT0001 complete [11214.543850] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:03:55 (1788937435) [11225.081965] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:04:05 (1788937445) [11235.385518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11236.326314] LustreError: 347293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11236.366377] LustreError: 347293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11236.450466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11241.441678] LustreError: 347294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11244.161846] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11246.565926] LustreError: 347293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11251.699954] LustreError: 347294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11255.881035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11256.455217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11262.787702] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11272.308571] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11280.235885] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11289.792751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11290.306860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11297.697703] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11306.819069] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11316.695141] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11319.184088] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11321.137677] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11322.955958] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11325.028837] Lustre: Failing over lustre-MDT0000 [11325.168976] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) [11325.572668] Lustre: server umount lustre-MDT0000 complete [11327.462158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11327.465965] LustreError: 347294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11327.481681] Lustre: Skipped 1 previous similar message [11327.522760] LustreError: 347294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11337.697366] LustreError: 347293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11337.738742] LustreError: 347293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11343.841516] Lustre: 340769:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788937550/real 1788937550] req@0000000020908d40 x1875836653353472/t0(0) o400->MGC192.168.204.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788937566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11343.874941] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11347.061000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11353.405763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11353.523457] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11356.886635] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11358.701893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11358.735745] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11358.837925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11359.212202] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11371.829469] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11373.848598] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11379.173726] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11381.886309] Lustre: Failing over lustre-MDT0000 [11382.233726] Lustre: server umount lustre-MDT0000 complete [11382.549060] LustreError: 347289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11382.586367] LustreError: 347289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [11384.288785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11384.296456] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11384.316148] Lustre: Skipped 2 previous similar messages [11400.674922] Lustre: 340769:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788937607/real 1788937607] req@00000000ac2eb463 x1875836653376256/t0(0) o400->MGC192.168.204.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788937623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11400.716676] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11405.213835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11410.918680] LustreError: 340767:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a1d0da44 x1875836653382144/t0(0) o250->MGC192.168.204.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11411.145787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11411.192828] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11413.195300] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11416.564542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11416.574888] Lustre: Skipped 2 previous similar messages [11416.640124] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11416.715393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11416.959453] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11431.145317] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11433.839296] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11447.269257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11447.269577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11447.281922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11447.284836] Lustre: Skipped 3 previous similar messages [11448.786234] Lustre: server umount lustre-MDT0000 complete [11452.386472] LustreError: 348715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11452.417495] LustreError: 348715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [11453.502684] LustreError: 349137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937676 with bad export cookie 10826925217412343039 [11453.518178] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11453.526431] LustreError: 349137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11454.032896] Lustre: server umount lustre-MDT0001 complete [11468.480196] Lustre: server umount lustre-OST0000 complete [11489.142904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11490.137974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11497.140490] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11510.282471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11510.752793] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11510.770500] Lustre: Skipped 2 previous similar messages [11516.612266] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11526.887856] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11534.877773] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11543.620708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11553.282257] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11560.380381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11562.642545] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11570.661526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11570.672314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11570.690050] Lustre: Skipped 1 previous similar message [11570.709304] Lustre: Skipped 1 previous similar message [11575.147900] Lustre: server umount lustre-OST0000 complete [11586.564489] Lustre: server umount lustre-MDT0000 complete [11590.117544] LustreError: 354558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11590.149043] LustreError: 354558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11591.846804] LustreError: 353619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937814 with bad export cookie 10826925217412343781 [11591.858890] LustreError: 353619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11591.864512] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11592.542904] Lustre: server umount lustre-MDT0001 complete [11608.971467] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [11613.789515] Key type lgssc unregistered [11614.096568] LNet: 357177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11614.100892] LNetError: 357177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11614.110544] LNet: Removed LNI 192.168.204.135@tcp [11615.358825] Key type .llcrypt unregistered [11615.374700] Key type ._llcrypt unregistered [11652.810589] Key type ._llcrypt registered [11652.814816] Key type .llcrypt registered [11652.967107] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [11654.522862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11654.711438] alg: No test for adler32 (adler32-zlib) [11655.940317] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [11656.321441] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [11658.143183] Key type lgssc registered [11659.687449] Lustre: Echo OBD driver; http://www.lustre.org/ [11669.666512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11669.689229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11671.170820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11676.847516] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11681.761218] LustreError: 358697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11681.782985] LustreError: 358697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11686.427520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11686.839946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11691.274508] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11698.877947] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11705.537891] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11712.400646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11712.808136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11716.921111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11720.256986] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11728.408979] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11746.787714] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11746.794220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11746.818227] Lustre: Skipped 1 previous similar message [11746.835532] Lustre: Skipped 2 previous similar messages [11750.890817] Lustre: server umount lustre-MDT0000 complete [11751.911152] LustreError: 360467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11751.927596] LustreError: 360467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11754.969940] LustreError: 358677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937977 with bad export cookie 606195516515918966 [11754.986149] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11754.988800] LustreError: 358677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11755.410354] Lustre: server umount lustre-MDT0001 complete [11770.132566] Lustre: server umount lustre-OST0000 complete [11786.295665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11786.890373] LustreError: 362433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11787.010767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11792.021438] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11792.363884] LustreError: 362434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11796.454567] LustreError: 362434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.447343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11802.000274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11806.724937] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11816.134972] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11823.952326] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11831.845621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11832.329396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11839.380404] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11848.511502] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11856.889191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11873.768450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11873.775268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11873.793848] Lustre: Skipped 1 previous similar message [11873.800431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11877.348798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11877.353586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11878.662484] Lustre: server umount lustre-MDT0000 complete [11882.466204] LustreError: 362433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11882.491730] LustreError: 362433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11883.085157] LustreError: 362415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938106 with bad export cookie 606195516515920079 [11883.087121] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11883.100066] LustreError: 362415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11883.459477] Lustre: server umount lustre-MDT0001 complete [11898.156529] Lustre: server umount lustre-OST0000 complete [11914.815835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11915.372691] LustreError: 366366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.446726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11920.314343] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11928.747889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11933.809551] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11940.922678] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11946.421448] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11953.474423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11954.000656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11954.007162] Lustre: Skipped 1 previous similar message [11961.222014] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11961.341715] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [11968.530269] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11975.648188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11975.659528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11975.670201] Lustre: Skipped 1 previous similar message [11975.674186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11975.678216] Lustre: Skipped 2 previous similar messages [11976.105278] Lustre: server umount lustre-OST0000 complete [11980.257259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11980.267271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11980.293052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11991.015990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11991.021805] Lustre: Skipped 4 previous similar messages [11991.171129] Lustre: server umount lustre-MDT0000 complete [11994.791896] LustreError: 366347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938217 with bad export cookie 606195516515921094 [11994.793594] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11994.806957] LustreError: 366347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11995.210925] Lustre: server umount lustre-MDT0001 complete [12008.965645] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [12011.698901] Key type lgssc unregistered [12011.999384] LNet: 369904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12012.011797] LNetError: 369904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12012.025364] LNet: Removed LNI 192.168.204.135@tcp [12013.049431] Key type .llcrypt unregistered [12013.052918] Key type ._llcrypt unregistered [12044.191461] Key type ._llcrypt registered [12044.198581] Key type .llcrypt registered [12044.338284] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [12045.837851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12045.890985] alg: No test for adler32 (adler32-zlib) [12047.145606] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [12047.520801] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [12049.295161] Key type lgssc registered [12050.954697] Lustre: Echo OBD driver; http://www.lustre.org/ [12060.292387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12060.307165] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12060.315917] LustreError: 371348:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12060.370982] LustreError: 371348:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12060.387892] LustreError: 371348:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12060.397340] LustreError: 371348:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12060.414660] LustreError: 371348:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12069.637224] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12069.651155] LustreError: 371747:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12069.684100] LustreError: 371747:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12069.700889] LustreError: 371747:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12069.716847] LustreError: 371747:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12069.725524] LustreError: 371747:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12091.326151] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [12094.053326] Key type lgssc unregistered [12094.349058] LNet: 372753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12094.359516] LNetError: 372753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12094.386192] LNet: Removed LNI 192.168.204.135@tcp [12095.164746] Key type .llcrypt unregistered [12095.167257] Key type ._llcrypt unregistered [12113.163674] Key type ._llcrypt registered [12113.165088] Key type .llcrypt registered [12113.292684] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [12128.676586] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [12129.984464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12130.068590] alg: No test for adler32 (adler32-zlib) [12131.254586] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [12131.601642] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [12133.343198] Key type lgssc registered [12134.518255] Lustre: Echo OBD driver; http://www.lustre.org/ [12176.872521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12176.928832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12178.451637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12178.496245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12178.690546] Lustre: lustre-MDT0000: new disk, initializing [12178.761278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12178.782671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12185.278727] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12198.699956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12198.994508] Lustre: 375916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12199.125687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12199.140726] Lustre: Skipped 1 previous similar message [12199.403962] Lustre: lustre-MDT0001: new disk, initializing [12199.682931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12199.777303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12199.798134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12205.988618] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12217.393679] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12223.893946] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12230.900785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12231.165519] Lustre: lustre-OST0000: new disk, initializing [12231.168665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12231.174821] Lustre: 377888:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12231.267340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12238.077679] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12238.472463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12238.489824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12238.646065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12248.520654] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12253.282299] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12253.540504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12259.276034] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12259.550645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12264.417818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12264.430251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12264.436105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12264.453143] Lustre: Skipped 1 previous similar message [12264.472155] Lustre: Skipped 1 previous similar message [12269.543116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12274.661217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12274.676141] Lustre: Skipped 2 previous similar messages [12276.193547] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12276.560316] Lustre: server umount lustre-OST0000 complete [12281.831946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12281.852946] LustreError: Skipped 1 previous similar message [12281.857639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12281.889934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12284.900510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12290.016983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12290.020847] Lustre: Skipped 3 previous similar messages [12295.136788] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12295.332191] Lustre: server umount lustre-MDT0000 complete [12299.486304] LustreError: 375907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938522 with bad export cookie 8073778991538119908 [12299.496815] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12299.500618] LustreError: 375907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12299.983747] Lustre: server umount lustre-MDT0001 complete [12309.322936] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12311.545629] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:22:12 (1788938532) [12318.935823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12319.384501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12324.174867] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12324.834206] LustreError: 380350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12324.856576] LustreError: 380350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12329.969203] LustreError: 380349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12333.119480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12333.580449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12338.181981] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12345.382620] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12350.582573] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12357.138500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12357.474344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12363.319651] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12370.347785] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12376.728993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12376.954186] Lustre: lustre-OST0001: new disk, initializing [12376.958782] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12376.964087] Lustre: 383188:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12377.064321] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12383.618277] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12386.442566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12386.458976] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12386.608152] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12393.450488] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12400.386087] Lustre: lustre-OST0001: Not available for connect from 192.168.204.35@tcp (stopping) [12400.399560] Lustre: Skipped 2 previous similar messages [12402.173273] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12402.175222] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12402.186035] LustreError: Skipped 1 previous similar message [12402.214327] Lustre: Skipped 1 previous similar message [12404.304851] Lustre: server umount lustre-OST0001 complete [12405.452799] LustreError: 383078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12412.391495] LustreError: 382171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12412.413276] LustreError: 382171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12417.510482] LustreError: 382172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12417.533193] LustreError: 382172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12425.188485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12425.208108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12425.230888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12425.239284] Lustre: Skipped 2 previous similar messages [12427.767062] LustreError: 382172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12427.803889] LustreError: 382172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12436.965050] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12437.251888] Lustre: server umount lustre-OST0000 complete [12442.082175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12442.100684] Lustre: Skipped 2 previous similar messages [12448.235208] LustreError: 380344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12448.246859] LustreError: 380344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12450.228887] Lustre: server umount lustre-MDT0000 complete [12454.469246] LustreError: 380329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938677 with bad export cookie 8073778991538120790 [12454.471314] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12454.487224] LustreError: 380329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12461.089494] Lustre: server umount lustre-MDT0001 complete [12474.539350] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [12477.356319] Key type lgssc unregistered [12477.656541] LNet: 385179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12477.662786] LNetError: 385179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12477.691995] LNet: Removed LNI 192.168.204.135@tcp [12478.779287] Key type .llcrypt unregistered [12478.781990] Key type ._llcrypt unregistered [12530.445092] Key type ._llcrypt registered [12530.448187] Key type .llcrypt registered [12530.610617] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [12532.090285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12532.120794] alg: No test for adler32 (adler32-zlib) [12533.252425] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [12533.588600] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [12535.271182] Key type lgssc registered [12536.830202] Lustre: Echo OBD driver; http://www.lustre.org/ [12545.294539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12545.308496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12546.672573] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12546.741256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12547.042488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12552.331797] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12561.953309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12562.035529] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12562.067518] Lustre: 387976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12562.462224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12567.833524] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12576.387260] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12582.647327] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12584.928394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12584.930121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12584.947797] Lustre: Skipped 1 previous similar message [12590.048925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12590.059201] Lustre: Skipped 2 previous similar messages [12595.170320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12595.188479] Lustre: Skipped 1 previous similar message [12599.263137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12599.488785] Lustre: server umount lustre-MDT0000 complete [12604.673071] LustreError: 388719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938827 with bad export cookie 4400417154874511253 [12604.675860] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12604.694266] LustreError: 388719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12605.278770] Lustre: server umount lustre-MDT0001 complete [12618.262877] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:27:19 (1788938839) [12626.629916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12627.217236] LustreError: 390490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12627.246529] LustreError: 390490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12627.288042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12632.553467] LustreError: 390491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12632.735865] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12637.668137] LustreError: 390490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12642.784762] LustreError: 390491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12643.541431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12644.015691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12649.269360] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12658.105863] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12664.667952] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12672.296387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12672.654430] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12672.896496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12680.554731] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12688.863829] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12697.896444] Lustre: Permanently deactivating lustre-OST0000 [12697.910059] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12706.905362] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12709.345925] LustreError: 393480:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12709.367046] LustreError: 393480:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12709.375268] LustreError: 393480:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12713.208141] LustreError: 393528:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12713.212380] LustreError: 393528:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12721.778549] Lustre: server umount lustre-OST0000 complete [12725.033945] LustreError: 393778:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12725.044654] LustreError: 393778:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12729.320647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12729.328290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12729.337498] Lustre: Skipped 1 previous similar message [12734.449376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12734.459653] Lustre: Skipped 2 previous similar messages [12737.589663] Lustre: server umount lustre-MDT0000 complete [12739.552157] LustreError: 390491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12739.579894] LustreError: 390491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12741.649919] LustreError: 390471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938964 with bad export cookie 4400417154874511834 [12741.655414] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12741.662731] LustreError: 390471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12752.116736] Lustre: server umount lustre-MDT0001 complete [12761.296399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12767.005255] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12769.827186] LustreError: 394937:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12777.097092] Lustre: 395277:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.135@tcp [12779.735415] Lustre: server umount MGS complete [12792.858756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12793.337074] LustreError: 396004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12793.359031] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12793.367303] Lustre: Skipped 1 previous similar message [12793.467746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12798.315386] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12807.143164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12807.399658] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12812.055453] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12820.687651] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12827.052212] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12829.552364] Lustre: Permanently reactivating lustre-OST0000 [12829.565425] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12829.574399] Lustre: Skipped 2 previous similar messages [12859.875823] LustreError: 397833:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12870.366279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12871.204596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12871.221799] Lustre: Skipped 1 previous similar message [12880.135307] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [12890.655151] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12899.813617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12899.828148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12899.853919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12900.838025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12903.529572] Lustre: server umount lustre-OST0000 complete [12910.048965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12910.060717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12910.086490] Lustre: Skipped 1 previous similar message [12915.755072] Lustre: server umount lustre-MDT0000 complete [12916.200624] LustreError: 397353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12916.233693] LustreError: 397353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12920.103737] LustreError: 395986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939143 with bad export cookie 4400417154874512884 [12920.111790] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12920.120818] LustreError: 395986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12920.709566] Lustre: server umount lustre-MDT0001 complete [12934.856884] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [12938.271527] Key type lgssc unregistered [12938.736901] LNet: 400372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12938.768552] LNetError: 400372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12938.802715] LNet: Removed LNI 192.168.204.135@tcp [12940.041503] Key type .llcrypt unregistered [12940.046072] Key type ._llcrypt unregistered [12962.087263] Key type ._llcrypt registered [12962.090607] Key type .llcrypt registered [12962.263810] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [12982.604670] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [12983.712914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12984.037965] alg: No test for adler32 (adler32-zlib) [12985.660867] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [12986.086577] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [12987.815280] Key type lgssc registered [12989.821972] Lustre: Echo OBD driver; http://www.lustre.org/ [13036.649700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13036.679562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13038.005824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13038.046665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13038.179214] Lustre: lustre-MDT0000: new disk, initializing [13038.278868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13038.297149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13043.660670] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13056.998537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13057.167665] Lustre: 403532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13057.250887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13057.256944] Lustre: Skipped 1 previous similar message [13057.367252] Lustre: lustre-MDT0001: new disk, initializing [13057.573768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13057.655914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13057.682056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13064.309600] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13076.995394] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13084.462435] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13092.862920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13093.311105] Lustre: lustre-OST0000: new disk, initializing [13093.319993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13093.329269] Lustre: 405502:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13093.475578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13100.107972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13100.133935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13100.357742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13102.775736] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13114.988508] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13121.707349] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13122.021262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13128.817876] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13129.079783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13136.363472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13136.370982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13136.371225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13136.385579] Lustre: Skipped 1 previous similar message [13137.675942] Lustre: server umount lustre-OST0000 complete [13144.545706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13144.568353] LustreError: Skipped 1 previous similar message [13144.570563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13144.593888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13144.600658] Lustre: Skipped 1 previous similar message [13146.594567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13146.595397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13148.530622] Lustre: server umount lustre-MDT0000 complete [13153.329585] LustreError: 403524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939376 with bad export cookie 11415954745258877317 [13153.337343] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13153.349060] LustreError: 403524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13153.784392] Lustre: server umount lustre-MDT0001 complete [13164.546856] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13166.854443] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:36:27 (1788939387) [13175.341388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13176.203741] LustreError: 407965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13176.230942] LustreError: 407965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13176.389015] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13181.414236] LustreError: 407966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13184.255465] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13186.541604] LustreError: 407965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13191.651300] LustreError: 407966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13196.048377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13196.427301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13202.215670] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13211.638090] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13220.259824] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13231.227749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13232.167331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13241.496518] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13251.836724] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13254.448972] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13257.595189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13267.936694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13267.946493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13267.967582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13267.978240] Lustre: Skipped 1 previous similar message [13267.997675] Lustre: Skipped 2 previous similar messages [13271.237833] Lustre: server umount lustre-OST0000 complete [13278.186257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13278.201781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13278.204200] LustreError: Skipped 1 previous similar message [13278.207737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13278.242481] Lustre: Skipped 1 previous similar message [13284.430588] Lustre: server umount lustre-MDT0000 complete [13288.416272] LustreError: 407966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13288.449486] LustreError: 407966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13290.717899] LustreError: 407946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939513 with bad export cookie 11415954745258878192 [13290.740171] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13291.470966] Lustre: server umount lustre-MDT0001 complete [13308.481291] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13312.062383] Key type lgssc unregistered [13312.672443] LNet: 411699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13312.695940] LNetError: 411699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13312.739780] LNet: Removed LNI 192.168.204.135@tcp [13314.279684] Key type .llcrypt unregistered [13314.282184] Key type ._llcrypt unregistered [13345.170439] Key type ._llcrypt registered [13345.177779] Key type .llcrypt registered [13345.375090] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13345.826330] Key type .llcrypt unregistered [13345.831892] Key type ._llcrypt unregistered [13367.383906] Key type ._llcrypt registered [13367.385849] Key type .llcrypt registered [13367.503820] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [13369.353038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13369.478060] alg: No test for adler32 (adler32-zlib) [13370.697913] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [13370.953283] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [13372.671587] Key type lgssc registered [13374.158583] Lustre: Echo OBD driver; http://www.lustre.org/ [13383.017400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13383.037678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13384.466917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13389.297307] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13394.919201] LustreError: 413920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13394.958961] LustreError: 413920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13398.743209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13399.147398] LustreError: 413919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13399.209913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13404.644967] LustreError: 413920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13404.666426] LustreError: 413920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13405.912542] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13409.763097] LustreError: 413916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13409.798462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13409.812048] LustreError: 413916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13414.881820] LustreError: 413919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13414.906214] LustreError: 413919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13415.001749] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13424.038698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13424.450609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13429.932319] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13443.013800] Lustre: server umount lustre-OST0000 complete [13447.103611] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13447.111593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13447.135462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13450.208480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13450.213081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13455.328675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13455.332810] Lustre: Skipped 1 previous similar message [13457.467573] Lustre: server umount lustre-MDT0000 complete [13460.450384] LustreError: 413919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13460.465796] LustreError: 413919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13461.472169] Lustre: 413228:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788939668/real 1788939668] req@000000001a1f7f6f x1875839122698240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788939684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13461.479039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13461.488070] LustreError: 413900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939684 with bad export cookie 4878502335670319477 [13461.492206] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13461.494480] Lustre: 413228:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [13461.846202] Lustre: server umount lustre-MDT0001 complete [13474.141675] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13477.243773] Key type lgssc unregistered [13477.484373] LNet: 417071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13477.491279] LNetError: 417071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13477.507429] LNet: Removed LNI 192.168.204.135@tcp [13478.709370] Key type .llcrypt unregistered [13478.711113] Key type ._llcrypt unregistered [13508.579929] Key type ._llcrypt registered [13508.581911] Key type .llcrypt registered [13508.714116] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [13510.753310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13510.852431] alg: No test for adler32 (adler32-zlib) [13512.250875] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [13512.731245] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [13514.419037] Key type lgssc registered [13515.863722] Lustre: Echo OBD driver; http://www.lustre.org/ [13525.960789] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13525.979809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13541.535459] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13546.657429] LustreError: 418464:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [13573.407520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13582.264993] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13591.856588] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13603.533785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13604.607642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13604.705508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13611.432212] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13614.052553] LustreError: 419505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13614.090407] LustreError: 419505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13619.169411] LustreError: 418490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13619.198597] LustreError: 418490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13623.272534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13623.986846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13631.686474] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13642.103273] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13650.125626] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13660.131397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13660.174538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13665.249782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13665.265404] Lustre: Skipped 1 previous similar message [13666.065730] Lustre: server umount lustre-OST0000 complete [13671.376435] Lustre: server umount lustre-MDT0000 complete [13674.979134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13674.996191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13675.013165] Lustre: Skipped 1 previous similar message [13675.040405] LustreError: 419508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13676.081307] LustreError: 418470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939899 with bad export cookie 16222901553773573302 [13676.109098] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13676.712711] Lustre: server umount lustre-MDT0001 complete [13688.439206] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13691.635981] Key type lgssc unregistered [13692.098616] LNet: 422038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13692.116561] LNetError: 422038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13692.145795] LNet: Removed LNI 192.168.204.135@tcp [13693.212837] Key type .llcrypt unregistered [13693.216760] Key type ._llcrypt unregistered [13724.416551] Key type ._llcrypt registered [13724.423749] Key type .llcrypt registered [13724.685711] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [13726.542868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13726.597614] alg: No test for adler32 (adler32-zlib) [13727.895406] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [13728.227339] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [13729.968150] Key type lgssc registered [13731.528451] Lustre: Echo OBD driver; http://www.lustre.org/ [13740.082866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13740.123871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13742.069718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13748.460583] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13751.273641] LustreError: 423457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13751.328187] LustreError: 423457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13756.392320] LustreError: 423458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13761.507800] LustreError: 423457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13762.116168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13763.158384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13770.651130] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13779.434758] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13785.703644] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13793.738316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13794.096149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13801.309066] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13809.569783] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13811.583808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13819.363900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13819.375440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13819.402745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13820.648290] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) [13821.922522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13821.926906] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13821.938811] Lustre: Skipped 1 previous similar message [13821.960171] Lustre: Skipped 1 previous similar message [13824.276867] Lustre: server umount lustre-MDT0000 complete [13825.757479] LustreError: 424470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13827.001415] LustreError: 423439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.35@tcp arrived at 1788940050 with bad export cookie 5250556236937460131 [13827.024029] LustreError: 423439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13829.189738] LustreError: 423437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940052 with bad export cookie 5250556236937459466 [13829.201211] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13832.170406] LustreError: 423458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13832.175699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13832.202220] LustreError: 423458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13832.245932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13832.262954] Lustre: Skipped 1 previous similar message [13835.904416] Lustre: server umount lustre-OST0000 complete [13844.458276] LustreError: 423457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13844.493400] LustreError: 423457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13846.355738] Lustre: server umount lustre-MDT0001 complete [13865.675338] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [13871.607135] Key type lgssc unregistered [13872.099627] LNet: 427173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13872.118582] LNetError: 427173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13872.157357] LNet: Removed LNI 192.168.204.135@tcp [13873.815298] Key type .llcrypt unregistered [13873.820272] Key type ._llcrypt unregistered [13908.573333] Key type ._llcrypt registered [13908.574765] Key type .llcrypt registered [13908.650794] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [13910.072579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13910.097298] alg: No test for adler32 (adler32-zlib) [13911.137752] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [13911.473296] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [13913.232723] Key type lgssc registered [13914.694527] Lustre: Echo OBD driver; http://www.lustre.org/ [13925.845983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13925.896413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13928.078410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13935.046678] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13937.635457] LustreError: 428594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13937.686848] LustreError: 428594:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13942.753641] LustreError: 428593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13946.861372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13947.335431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13953.095135] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13962.967856] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13971.510412] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13980.507984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13981.151459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13990.045682] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [13997.512601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [14000.994988] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14007.785369] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14007.809561] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14007.821967] Lustre: Skipped 1 previous similar message [14008.571847] Lustre: server umount lustre-MDT0001 complete [14012.906340] LustreError: 428593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14012.939554] LustreError: 428593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14018.030294] LustreError: 428586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14018.072831] LustreError: 428586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14023.136410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14023.139493] LustreError: 430608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14023.147903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14023.147912] Lustre: Skipped 1 previous similar message [14023.168685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14023.203882] Lustre: Skipped 1 previous similar message [14024.946653] Lustre: server umount lustre-OST0000 complete [14032.144132] Lustre: server umount lustre-MDT0000 complete [14048.357910] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [14051.918272] Key type lgssc unregistered [14052.288969] LNet: 432284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14052.300433] LNetError: 432284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14052.315839] LNet: Removed LNI 192.168.204.135@tcp [14053.734593] Key type .llcrypt unregistered [14053.739992] Key type ._llcrypt unregistered [14092.923431] Key type ._llcrypt registered [14092.928071] Key type .llcrypt registered [14093.122875] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [14113.076907] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [14114.551993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14114.564388] alg: No test for adler32 (adler32-zlib) [14116.399541] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [14116.939598] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [14118.743191] Key type lgssc registered [14120.895318] Lustre: Echo OBD driver; http://www.lustre.org/ [14177.062617] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [14201.625190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14201.716206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14203.294374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14203.349230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14203.409915] Lustre: lustre-MDT0000: new disk, initializing [14203.525619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14203.550596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14209.257060] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [14228.596037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14228.900880] Lustre: lustre-OST0000: new disk, initializing [14228.914219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14228.928362] Lustre: Skipped 1 previous similar message [14228.944525] Lustre: 437348:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14229.131922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14236.954500] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [14237.823323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14237.832843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14237.870179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14248.485362] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14267.635589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14267.825181] Lustre: 436329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14267.922276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14268.042374] Lustre: lustre-MDT0001: new disk, initializing [14268.231851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14268.293059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14268.312479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14274.785313] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [14285.719461] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14285.941672] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14291.680614] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14292.092252] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14302.179213] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14302.196895] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14302.203538] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14302.210959] Lustre: Skipped 1 previous similar message [14302.224485] Lustre: Skipped 1 previous similar message [14305.120451] Lustre: server umount lustre-MDT0001 complete [14312.420033] LustreError: 436334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14312.466933] LustreError: 436334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [14317.552934] LustreError: 437356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14317.584223] LustreError: 437356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14322.675921] LustreError: 436334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14322.718200] LustreError: 436334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14327.778789] LustreError: 436340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14327.779282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14327.848697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14327.860924] Lustre: Skipped 1 previous similar message [14332.332434] Lustre: server umount lustre-OST0000 complete [14332.925990] LustreError: 436339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14352.351162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14352.722367] Lustre: server umount lustre-MDT0000 complete [14369.817343] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [14372.972341] Key type lgssc unregistered [14373.303215] LNet: 441101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14373.311616] LNetError: 441101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14373.328855] LNet: Removed LNI 192.168.204.135@tcp [14374.592962] Key type .llcrypt unregistered [14374.594590] Key type ._llcrypt unregistered [14430.448400] Key type ._llcrypt registered [14430.450187] Key type .llcrypt registered [14430.586942] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [14431.091685] Key type .llcrypt unregistered [14431.098363] Key type ._llcrypt unregistered [14458.891656] Key type ._llcrypt registered [14458.899278] Key type .llcrypt registered [14459.117797] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [14460.604487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14460.663059] alg: No test for adler32 (adler32-zlib) [14462.060940] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [14462.471405] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [14464.311240] Key type lgssc registered [14465.768417] Lustre: Echo OBD driver; http://www.lustre.org/ [14497.173593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14497.205815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14498.660863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14498.697646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14498.781673] Lustre: lustre-MDT0000: new disk, initializing [14498.880393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14498.905340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14504.788417] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [14517.595340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14517.757568] Lustre: 444490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14517.833108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14517.836964] Lustre: Skipped 1 previous similar message [14517.958294] Lustre: lustre-MDT0001: new disk, initializing [14518.057038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14518.118620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14518.136236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14523.874865] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [14534.699813] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14541.042766] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14549.181888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14549.480745] Lustre: lustre-OST0000: new disk, initializing [14549.486783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14549.496698] Lustre: 446459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14549.605917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14556.207060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14556.215126] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14556.375433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14556.828313] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [14568.112904] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14577.127216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14577.129404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14577.169358] Lustre: Skipped 1 previous similar message [14577.172386] Lustre: Skipped 2 previous similar messages [14581.600299] Lustre: server umount lustre-MDT0000 complete [14586.295299] LustreError: 444479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940809 with bad export cookie 11495857815207008854 [14586.297202] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14586.316385] LustreError: 444479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14586.789478] Lustre: server umount lustre-MDT0001 complete [14602.914348] Lustre: server umount lustre-OST0000 complete [14617.459994] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:00:38 (1788940838) [14627.806166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14628.555063] LustreError: 448488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14628.589738] LustreError: 448488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14628.729060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14633.961924] LustreError: 448489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14634.578101] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [14638.058913] LustreError: 448488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14643.170502] LustreError: 448489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14646.727646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14647.168408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14652.425640] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [14662.033612] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14668.680498] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14689.518243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14689.731176] Lustre: Found index 0 for lustre-OST0000, updating log [14689.887360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14698.072566] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [14698.103779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14708.283433] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14718.960153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14718.960174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14718.990263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14719.013861] Lustre: Skipped 1 previous similar message [14720.034144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14720.052525] Lustre: Skipped 1 previous similar message [14720.895452] Lustre: server umount lustre-OST0000 complete [14741.483983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14741.501738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14741.507023] Lustre: Skipped 2 previous similar messages [14741.518143] Lustre: Skipped 2 previous similar messages [14746.595947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14746.622272] Lustre: Skipped 1 previous similar message [14750.303492] LNet: 1 local NIs in recovery (showing 1): 192.168.204.135@tcp [14754.271273] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14754.534474] Lustre: server umount lustre-MDT0000 complete [14758.880727] LustreError: 449227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14758.919610] LustreError: 449227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14759.756228] LustreError: 448470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940982 with bad export cookie 11495857815207010254 [14759.769990] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14759.782034] LustreError: 448470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14760.617589] Lustre: server umount lustre-MDT0001 complete [14771.213235] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:03:12 (1788940992) [15405.476024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15406.145752] LustreError: 454919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15406.224590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15411.549261] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [15411.681504] LustreError: 454920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15416.801985] LustreError: 454919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15421.278681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15421.667938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15427.272922] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [15435.618719] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15441.710646] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15462.840721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15463.333541] Lustre: Found index 0 for lustre-OST0000, updating log [15463.869467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15467.232127] LNet: 1 local NIs in recovery (showing 1): 192.168.204.135@tcp [15472.328811] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [15483.524315] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15486.698135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15486.727545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15486.737320] Lustre: Skipped 6 previous similar messages [15491.934303] Lustre: server umount lustre-OST0000 complete [15496.801996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15496.834409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15496.843806] Lustre: Skipped 1 previous similar message [15498.928187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15498.932394] Lustre: Skipped 2 previous similar messages [15502.299478] Lustre: server umount lustre-MDT0000 complete [15503.009175] LustreError: 454916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15503.033189] LustreError: 454916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15506.161620] LustreError: 456769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941729 with bad export cookie 11495857815207011339 [15506.164832] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15506.171464] LustreError: 456769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15506.612343] Lustre: server umount lustre-MDT0001 complete [15527.647159] LNet: 1 local NIs in recovery (showing 1): 192.168.204.135@tcp [15537.975194] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [15547.878669] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [15587.324382] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:16:48 (1788941808) [15605.746233] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [15608.415226] Key type lgssc unregistered [15608.743632] LNet: 461535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15608.767224] LNetError: 461535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15726.566782] LNet: Removed LNI 192.168.204.135@tcp [15727.674570] Key type .llcrypt unregistered [15727.678929] Key type ._llcrypt unregistered [15745.801689] Key type ._llcrypt registered [15745.803902] Key type .llcrypt registered [15745.988786] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [15747.551491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15747.585605] alg: No test for adler32 (adler32-zlib) [15748.693221] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [15748.958840] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [15750.703187] Key type lgssc registered [15752.123771] Lustre: Echo OBD driver; http://www.lustre.org/ [15757.670911] LNet: Added LNI 192.168.204.135@tcp100 [8/256/0/180] [15758.662639] LNet: Added LNI 192.168.204.135@tcp101 [8/256/0/180] [15759.701868] LNet: Added LNI 192.168.204.135@tcp102 [8/256/0/180] [15760.850358] LNet: Added LNI 192.168.204.135@tcp103 [8/256/0/180] [15761.958274] LNet: Added LNI 192.168.204.135@tcp104 [8/256/0/180] [15763.115829] LNet: Added LNI 192.168.204.135@tcp105 [8/256/0/180] [15764.174448] LNet: Added LNI 192.168.204.135@tcp106 [8/256/0/180] [15765.204409] LNet: Added LNI 192.168.204.135@tcp107 [8/256/0/180] [15766.273470] LNet: Added LNI 192.168.204.135@tcp108 [8/256/0/180] [15767.297836] LNet: Added LNI 192.168.204.135@tcp109 [8/256/0/180] [15768.174478] LNet: Added LNI 192.168.204.135@tcp110 [8/256/0/180] [15769.116864] LNet: Added LNI 192.168.204.135@tcp111 [8/256/0/180] [15770.111824] LNet: Added LNI 192.168.204.135@tcp112 [8/256/0/180] [15771.297621] LNet: Added LNI 192.168.204.135@tcp113 [8/256/0/180] [15772.634689] LNet: Added LNI 192.168.204.135@tcp114 [8/256/0/180] [15773.749148] LNet: Added LNI 192.168.204.135@tcp115 [8/256/0/180] [15775.081907] LNet: Added LNI 192.168.204.135@tcp116 [8/256/0/180] [15776.263177] LNet: Added LNI 192.168.204.135@tcp117 [8/256/0/180] [15777.447863] LNet: Added LNI 192.168.204.135@tcp118 [8/256/0/180] [15778.491339] LNet: Added LNI 192.168.204.135@tcp119 [8/256/0/180] [15779.660280] LNet: Added LNI 192.168.204.135@tcp120 [8/256/0/180] [15780.794947] LNet: Added LNI 192.168.204.135@tcp121 [8/256/0/180] [15782.011615] LNet: Added LNI 192.168.204.135@tcp122 [8/256/0/180] [15783.035591] LNet: Added LNI 192.168.204.135@tcp123 [8/256/0/180] [15784.128513] LNet: Added LNI 192.168.204.135@tcp124 [8/256/0/180] [15785.281396] LNet: Added LNI 192.168.204.135@tcp125 [8/256/0/180] [15786.329564] LNet: Added LNI 192.168.204.135@tcp126 [8/256/0/180] [15787.525896] LNet: Added LNI 192.168.204.135@tcp127 [8/256/0/180] [15789.051544] LNet: Added LNI 192.168.204.135@tcp128 [8/256/0/180] [15790.303421] LNet: Added LNI 192.168.204.135@tcp129 [8/256/0/180] [15791.308533] LNet: Added LNI 192.168.204.135@tcp130 [8/256/0/180] [15792.551233] LNet: Added LNI 192.168.204.135@tcp131 [8/256/0/180] [15793.643356] LNet: Added LNI 192.168.204.135@tcp132 [8/256/0/180] [15794.605735] LNet: Added LNI 192.168.204.135@tcp133 [8/256/0/180] [15795.665614] LNet: Added LNI 192.168.204.135@tcp134 [8/256/0/180] [15797.009734] LNet: Added LNI 192.168.204.135@tcp135 [8/256/0/180] [15806.323253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15806.359466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15807.495285] Lustre: 464570:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.135@tcp: can't add NID 192.168.204.135@tcp130: rc = -75 [15807.564475] LNetError: 464598:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.135 [15807.577186] LNetError: Protocol error connecting to 192.168.204.135@tcp114 at host 192.168.204.135:988: is it running a compatible version of Lustre? [15807.595583] LNetError: Skipped 4 previous similar messages [15807.706782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15807.766412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15807.879118] Lustre: lustre-MDT0000: new disk, initializing [15807.981222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15807.999334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15808.416153] LNet: 6 local NIs in recovery (showing 6): 192.168.204.135@tcp121, 192.168.204.135@tcp125, 192.168.204.135@tcp126, 192.168.204.135@tcp124, 192.168.204.135@tcp128, 192.168.204.135@tcp129 [15808.417125] LNetError: 464596:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.135 [15808.457395] LNetError: 464596:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [15808.479185] LNetError: Protocol error connecting to 192.168.204.135@tcp105 at host 192.168.204.135:988: is it running a compatible version of Lustre? [15808.499413] LNetError: Skipped 6 previous similar messages [15812.908995] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [15824.436270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15824.629641] Lustre: 464586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15824.683090] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15824.686525] Lustre: Skipped 1 previous similar message [15824.756297] Lustre: lustre-MDT0001: new disk, initializing [15824.789636] Lustre: 465389:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.135@tcp: can't add NID 192.168.204.135@tcp130: rc = -75 [15824.795260] Lustre: 465389:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15824.906629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15824.943100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15824.967964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15829.620758] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [15834.711664] Lustre: 465822:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.135@tcp: can't add NID 192.168.204.135@tcp130: rc = -75 [15834.721824] Lustre: 465822:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15837.026067] LNetError: 462030:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.135 [15837.040634] LNetError: 462030:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15837.048575] LNetError: Protocol error connecting to 192.168.204.135@tcp117 at host 192.168.204.135:988: is it running a compatible version of Lustre? [15839.312696] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15840.115797] LNetError: 464604:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.135 [15840.134878] LNetError: 464604:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15840.141509] LNetError: Protocol error connecting to 192.168.204.135@tcp123 at host 192.168.204.135:988: is it running a compatible version of Lustre? [15840.150093] LNetError: Skipped 1 previous similar message [15845.249748] LNetError: 464598:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.135 [15845.261118] LNetError: 464598:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15845.269032] LNetError: Protocol error connecting to 192.168.204.135@tcp129 at host 192.168.204.135:988: is it running a compatible version of Lustre? [15845.275698] LNetError: Skipped 2 previous similar messages [15845.326451] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15852.582536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15852.953400] Lustre: lustre-OST0000: new disk, initializing [15852.956599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15852.962501] Lustre: 466559:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15853.074447] Lustre: 466565:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.135@tcp: can't add NID 192.168.204.135@tcp130: rc = -75 [15853.090695] Lustre: 466565:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15853.137653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15853.416261] LNetError: 462030:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.135 [15853.425993] LNetError: 462030:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [15853.437239] LNetError: Protocol error connecting to 192.168.204.135@tcp123 at host 192.168.204.135:988: is it running a compatible version of Lustre? [15853.449649] LNetError: Skipped 5 previous similar messages [15859.760801] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [15860.835856] Lustre: 466970:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.135@tcp: can't add NID 192.168.204.135@tcp130: rc = -75 [15860.847706] Lustre: 466970:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15860.952746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15860.963397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15861.129730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15870.128754] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15871.849198] LNetError: 464590:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.135 [15871.863855] LNetError: 464590:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [15871.871142] LNetError: Protocol error connecting to 192.168.204.135@tcp129 at host 192.168.204.135:988: is it running a compatible version of Lustre? [15871.880507] LNetError: Skipped 11 previous similar messages [15881.697447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15881.708582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15881.721451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15881.724891] Lustre: Skipped 1 previous similar message [15884.629079] Lustre: server umount lustre-OST0000 complete [15891.424316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15891.435353] LustreError: Skipped 1 previous similar message [15891.438324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15891.451353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15891.456253] Lustre: Skipped 1 previous similar message [15895.392926] Lustre: server umount lustre-MDT0000 complete [15899.617141] LustreError: 464580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942122 with bad export cookie 17208265438181479517 [15899.641792] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15900.179420] Lustre: server umount lustre-MDT0001 complete [15904.610356] LNetError: 464591:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.135 [15904.634206] LNetError: 464591:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [15904.643792] LNetError: Protocol error connecting to 192.168.204.135@tcp123 at host 192.168.204.135:988: is it running a compatible version of Lustre? [15904.655203] LNetError: Skipped 14 previous similar messages [15912.542917] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [15915.875679] Key type lgssc unregistered [15916.316454] LNet: 468477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15916.325770] LNetError: 468477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15916.332996] LNet: Removed LNI 192.168.204.135@tcp [15917.344079] LNet: Removed LNI 192.168.204.135@tcp100 [15917.352987] LNetError: 468477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15917.368215] LNetError: 468477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15917.381496] LNet: Removed LNI 192.168.204.135@tcp101 [15918.431523] LNet: Removed LNI 192.168.204.135@tcp102 [15918.436216] LNetError: 468477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15918.454133] LNetError: 468477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15918.465841] LNet: Removed LNI 192.168.204.135@tcp103 [15919.519198] LNet: Removed LNI 192.168.204.135@tcp104 [15920.543179] LNet: Removed LNI 192.168.204.135@tcp105 [15920.556132] LNetError: 468477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15920.577753] LNetError: 468477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15920.598844] LNet: Removed LNI 192.168.204.135@tcp106 [15921.631806] LNet: Removed LNI 192.168.204.135@tcp107 [15922.655159] LNet: Removed LNI 192.168.204.135@tcp108 [15923.680244] LNet: Removed LNI 192.168.204.135@tcp109 [15924.703163] LNet: Removed LNI 192.168.204.135@tcp110 [15924.707344] LNetError: 468477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15924.718961] LNetError: 468477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15924.729992] LNet: Removed LNI 192.168.204.135@tcp111 [15925.792106] LNet: Removed LNI 192.168.204.135@tcp112 [15926.815242] LNet: Removed LNI 192.168.204.135@tcp113 [15927.839756] LNet: Removed LNI 192.168.204.135@tcp114 [15928.863206] LNet: Removed LNI 192.168.204.135@tcp115 [15929.887155] LNet: Removed LNI 192.168.204.135@tcp116 [15930.975582] LNet: Removed LNI 192.168.204.135@tcp117 [15932.000430] LNet: Removed LNI 192.168.204.135@tcp118 [15933.024122] LNet: Removed LNI 192.168.204.135@tcp119 [15933.038073] LNetError: 468477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15933.048044] LNetError: 468477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15933.059893] LNet: Removed LNI 192.168.204.135@tcp120 [15934.111179] LNet: Removed LNI 192.168.204.135@tcp121 [15935.136935] LNet: Removed LNI 192.168.204.135@tcp122 [15936.159645] LNet: Removed LNI 192.168.204.135@tcp123 [15937.183482] LNet: Removed LNI 192.168.204.135@tcp124 [15938.207208] LNet: Removed LNI 192.168.204.135@tcp125 [15939.231229] LNet: Removed LNI 192.168.204.135@tcp126 [15940.255330] LNet: Removed LNI 192.168.204.135@tcp127 [15941.343268] LNet: Removed LNI 192.168.204.135@tcp128 [15942.367362] LNet: Removed LNI 192.168.204.135@tcp129 [15942.369449] LNet: Removed LNI 192.168.204.135@tcp130 [15942.394499] LNet: Removed LNI 192.168.204.135@tcp131 [15942.401323] LNet: Removed LNI 192.168.204.135@tcp132 [15942.403852] LNet: Removed LNI 192.168.204.135@tcp133 [15942.408498] LNet: Removed LNI 192.168.204.135@tcp134 [15942.430434] LNet: Removed LNI 192.168.204.135@tcp135 [15943.378304] Key type .llcrypt unregistered [15943.380076] Key type ._llcrypt unregistered [15971.958827] Key type ._llcrypt registered [15971.961733] Key type .llcrypt registered [15972.100411] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [15988.539646] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [15990.213745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15990.324232] alg: No test for adler32 (adler32-zlib) [15991.438888] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [15991.713454] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [15993.455214] Key type lgssc registered [15994.968654] Lustre: Echo OBD driver; http://www.lustre.org/ [16037.754776] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:24:18 (1788942258) [16060.311418] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [16063.447637] Key type lgssc unregistered [16063.828502] LNet: 472571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16063.837960] LNetError: 472571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16063.857075] LNet: Removed LNI 192.168.204.135@tcp [16064.630718] Key type .llcrypt unregistered [16064.637819] Key type ._llcrypt unregistered [16087.017540] Key type ._llcrypt registered [16087.019163] Key type .llcrypt registered [16087.181876] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [16088.587417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16088.663155] alg: No test for adler32 (adler32-zlib) [16090.203295] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [16090.520679] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [16092.207842] Key type lgssc registered [16094.106557] Lustre: Echo OBD driver; http://www.lustre.org/ [16105.060856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16105.092506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16106.408494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16106.454713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16106.523338] Lustre: lustre-MDT0000: new disk, initializing [16106.614056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16106.643236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16112.055576] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [16123.838238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16123.975646] Lustre: 473786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16124.054334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16124.062284] Lustre: Skipped 1 previous similar message [16124.187874] Lustre: lustre-MDT0001: new disk, initializing [16124.314548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16124.345544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16124.356330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16129.291481] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [16138.793308] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16144.575098] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16151.899655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16152.125597] Lustre: lustre-OST0000: new disk, initializing [16152.131615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16152.141806] Lustre: 475754:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16152.214259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16158.357251] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [16161.436223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16161.443867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16161.588541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16168.080938] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16172.071774] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing lnet_if_list [16178.225588] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [16178.268206] LNet: Added LNI 192.168.204.135@tcp10 [8/256/0/180] [16184.781384] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [16184.808453] LNet: Added LNI 192.168.204.135@tcp11 [8/256/0/180] [16190.499554] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [16190.550625] LNet: Added LNI 192.168.204.135@tcp12 [8/256/0/180] [16195.834702] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [16195.878640] LNet: Added LNI 192.168.204.135@tcp13 [8/256/0/180] [16201.065813] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [16201.093585] LNet: Added LNI 192.168.204.135@tcp14 [8/256/0/180] [16207.374803] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [16207.447885] LNet: Added LNI 192.168.204.135@tcp15 [8/256/0/180] [16211.940528] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [16211.961551] LNet: Added LNI 192.168.204.135@tcp16 [8/256/0/180] [16216.463457] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [16216.511950] LNet: Added LNI 192.168.204.135@tcp17 [8/256/0/180] [16221.737536] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [16221.764670] LNet: Added LNI 192.168.204.135@tcp18 [8/256/0/180] [16227.191533] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [16227.222846] LNet: Added LNI 192.168.204.135@tcp19 [8/256/0/180] [16231.909863] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [16231.942471] LNet: Added LNI 192.168.204.135@tcp20 [8/256/0/180] [16237.325902] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16237.350372] LNet: Added LNI 192.168.204.135@tcp21 [8/256/0/180] [16242.837301] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16242.865886] LNet: Added LNI 192.168.204.135@tcp22 [8/256/0/180] [16248.005621] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16248.033744] LNet: Added LNI 192.168.204.135@tcp23 [8/256/0/180] [16253.769627] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16253.797163] LNet: Added LNI 192.168.204.135@tcp24 [8/256/0/180] [16260.101562] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16260.130065] LNet: Added LNI 192.168.204.135@tcp25 [8/256/0/180] [16266.016983] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16266.038966] LNet: Added LNI 192.168.204.135@tcp26 [8/256/0/180] [16272.807095] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16272.835676] LNet: Added LNI 192.168.204.135@tcp27 [8/256/0/180] [16279.646847] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16279.683089] LNet: Added LNI 192.168.204.135@tcp28 [8/256/0/180] [16286.055835] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16286.094989] LNet: Added LNI 192.168.204.135@tcp29 [8/256/0/180] [16293.495704] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16293.540094] LNet: Added LNI 192.168.204.135@tcp30 [8/256/0/180] [16300.498024] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16300.551270] LNet: Added LNI 192.168.204.135@tcp31 [8/256/0/180] [16307.581603] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16307.621190] LNet: Added LNI 192.168.204.135@tcp32 [8/256/0/180] [16313.765864] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16313.795945] LNet: Added LNI 192.168.204.135@tcp33 [8/256/0/180] [16320.936334] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16320.987825] LNet: Added LNI 192.168.204.135@tcp34 [8/256/0/180] [16328.057902] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16328.112640] LNet: Added LNI 192.168.204.135@tcp35 [8/256/0/180] [16335.799436] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16335.834293] LNet: Added LNI 192.168.204.135@tcp36 [8/256/0/180] [16342.472252] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16342.493695] LNet: Added LNI 192.168.204.135@tcp37 [8/256/0/180] [16349.801834] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16349.841788] LNet: Added LNI 192.168.204.135@tcp38 [8/256/0/180] [16358.089763] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16358.123486] LNet: Added LNI 192.168.204.135@tcp39 [8/256/0/180] [16366.591880] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16366.649563] LNet: Added LNI 192.168.204.135@tcp40 [8/256/0/180] [16374.640983] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16374.669368] LNet: Added LNI 192.168.204.135@tcp41 [8/256/0/180] [16382.849442] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16382.903889] LNet: Added LNI 192.168.204.135@tcp42 [8/256/0/180] [16391.161153] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16391.183432] LNet: Added LNI 192.168.204.135@tcp43 [8/256/0/180] [16399.976866] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16400.017846] LNet: Added LNI 192.168.204.135@tcp44 [8/256/0/180] [16408.570705] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16408.605047] LNet: Added LNI 192.168.204.135@tcp45 [8/256/0/180] [16416.552686] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16416.584232] LNet: Added LNI 192.168.204.135@tcp46 [8/256/0/180] [16424.648689] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16424.674526] LNet: Added LNI 192.168.204.135@tcp47 [8/256/0/180] [16432.972660] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16433.026343] LNet: Added LNI 192.168.204.135@tcp48 [8/256/0/180] [16440.660686] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16440.695214] LNet: Added LNI 192.168.204.135@tcp49 [8/256/0/180] [16448.510762] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16448.557522] LNet: Added LNI 192.168.204.135@tcp50 [8/256/0/180] [16472.041642] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16472.063130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16472.079671] Lustre: Skipped 2 previous similar messages [16472.098841] Lustre: Skipped 2 previous similar messages [16474.785554] Lustre: server umount lustre-MDT0000 complete [16479.015571] LustreError: 473776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942702 with bad export cookie 14298135601123517693 [16479.018312] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16479.024246] LustreError: 473776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16479.626224] Lustre: server umount lustre-MDT0001 complete [16498.661287] Lustre: 473058:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788942705/real 1788942705] req@0000000023292968 x1875841974970240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788942721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16498.717394] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16500.452764] Lustre: 473059:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788942707/real 1788942707] req@000000000b5d3f6b x1875841974970368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788942723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16502.365884] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [16503.775103] Lustre: 473057:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788942710/real 1788942710] req@000000004e9fb84e x1875841974970496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788942726 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16505.823131] Lustre: 473056:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788942712/real 1788942712] req@0000000024e10f9f x1875841974970752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788942728 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16510.948543] Lustre: 473058:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788942717/real 1788942717] req@000000009cf5f681 x1875841974970880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788942733 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16513.352823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16524.385482] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc66d2e930f556f30 [16524.407942] Lustre: MGC192.168.204.135@tcp: Connection restored to 0@lo (at 0@lo) [16524.648399] LustreError: 475748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16524.666849] LustreError: 475748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [16524.716231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16527.798198] LustreError: 475749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16527.804443] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16528.801931] LustreError: 475748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16529.761966] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [16533.986957] LustreError: 475749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16539.107616] LustreError: 475748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16539.356924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16539.846344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16543.999183] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16545.180909] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [16553.346863] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16562.513946] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16585.194973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16585.214415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16585.215972] Lustre: Skipped 1 previous similar message [16585.224904] Lustre: Skipped 1 previous similar message [16590.316561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16590.327842] Lustre: Skipped 1 previous similar message [16591.352222] Lustre: server umount lustre-OST0000 complete [16600.547631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16600.564488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16600.567719] Lustre: Skipped 1 previous similar message [16602.508345] Lustre: server umount lustre-MDT0000 complete [16605.665976] LustreError: 500702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16605.694395] LustreError: 500702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16606.061114] LustreError: 476964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942829 with bad export cookie 14298135601123520304 [16606.065344] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16606.071393] LustreError: 476964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16606.329683] Lustre: server umount lustre-MDT0001 complete [16616.280827] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [16618.737782] Key type lgssc unregistered [16619.072378] LNet: 502804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16619.078466] LNetError: 502804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16619.082156] LNet: Removed LNI 192.168.204.135@tcp [16619.084398] LNet: Removed LNI 192.168.204.135@tcp10 [16619.086626] LNet: Removed LNI 192.168.204.135@tcp11 [16619.088838] LNet: Removed LNI 192.168.204.135@tcp12 [16619.091013] LNet: Removed LNI 192.168.204.135@tcp13 [16619.093446] LNet: Removed LNI 192.168.204.135@tcp14 [16619.095333] LNet: Removed LNI 192.168.204.135@tcp15 [16619.098176] LNet: Removed LNI 192.168.204.135@tcp16 [16619.100453] LNet: Removed LNI 192.168.204.135@tcp17 [16619.103981] LNet: Removed LNI 192.168.204.135@tcp18 [16619.106950] LNet: Removed LNI 192.168.204.135@tcp19 [16619.111379] LNet: Removed LNI 192.168.204.135@tcp20 [16619.115818] LNet: Removed LNI 192.168.204.135@tcp21 [16619.119240] LNet: Removed LNI 192.168.204.135@tcp22 [16619.133371] LNet: Removed LNI 192.168.204.135@tcp23 [16619.135300] LNet: Removed LNI 192.168.204.135@tcp24 [16619.141930] LNet: Removed LNI 192.168.204.135@tcp25 [16619.144489] LNet: Removed LNI 192.168.204.135@tcp26 [16619.148584] LNet: Removed LNI 192.168.204.135@tcp27 [16619.150801] LNet: Removed LNI 192.168.204.135@tcp28 [16619.153320] LNet: Removed LNI 192.168.204.135@tcp29 [16619.155738] LNet: Removed LNI 192.168.204.135@tcp30 [16619.158664] LNet: Removed LNI 192.168.204.135@tcp31 [16619.161986] LNet: Removed LNI 192.168.204.135@tcp32 [16619.165676] LNet: Removed LNI 192.168.204.135@tcp33 [16619.168213] LNet: Removed LNI 192.168.204.135@tcp34 [16619.170683] LNet: Removed LNI 192.168.204.135@tcp35 [16619.173202] LNet: Removed LNI 192.168.204.135@tcp36 [16619.175668] LNet: Removed LNI 192.168.204.135@tcp37 [16619.177895] LNet: Removed LNI 192.168.204.135@tcp38 [16619.180588] LNet: Removed LNI 192.168.204.135@tcp39 [16619.183778] LNet: Removed LNI 192.168.204.135@tcp40 [16619.187712] LNet: Removed LNI 192.168.204.135@tcp41 [16619.190731] LNet: Removed LNI 192.168.204.135@tcp42 [16619.194487] LNet: Removed LNI 192.168.204.135@tcp43 [16619.197684] LNet: Removed LNI 192.168.204.135@tcp44 [16619.200421] LNet: Removed LNI 192.168.204.135@tcp45 [16619.202992] LNet: Removed LNI 192.168.204.135@tcp46 [16619.207776] LNet: Removed LNI 192.168.204.135@tcp47 [16619.210518] LNet: Removed LNI 192.168.204.135@tcp48 [16619.213669] LNet: Removed LNI 192.168.204.135@tcp49 [16619.226479] LNet: Removed LNI 192.168.204.135@tcp50 [16619.959736] Key type .llcrypt unregistered [16619.964795] Key type ._llcrypt unregistered [16638.459687] Key type ._llcrypt registered [16638.462382] Key type .llcrypt registered [16638.550337] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [16651.644313] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [16652.746098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16652.879053] alg: No test for adler32 (adler32-zlib) [16653.990577] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [16654.236458] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [16655.967192] Key type lgssc registered [16657.072076] Lustre: Echo OBD driver; http://www.lustre.org/ [16691.708218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16691.744802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16692.987838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16693.040697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16693.138377] Lustre: lustre-MDT0000: new disk, initializing [16693.232982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16693.256138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16697.754061] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [16708.650156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16708.791935] Lustre: 506002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16708.863605] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16708.871906] Lustre: Skipped 1 previous similar message [16708.889906] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [16709.008285] Lustre: lustre-MDT0001: new disk, initializing [16709.176426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16709.223310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16709.239144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16713.970171] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [16723.727305] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16729.942819] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16737.264269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16737.455918] Lustre: *** cfs_fail_loc=90f, val=0*** [16737.545892] Lustre: lustre-OST0000: new disk, initializing [16737.550919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16737.563845] Lustre: 508018:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16737.660500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16743.886770] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [16744.523666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16744.544097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16744.647565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16753.488162] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16765.413742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16765.416372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16765.448851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16765.459322] Lustre: Skipped 1 previous similar message [16766.782334] Lustre: server umount lustre-OST0000 complete [16775.651100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16775.651561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16775.666743] Lustre: Skipped 1 previous similar message [16775.671959] LustreError: Skipped 1 previous similar message [16775.675534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16775.686496] Lustre: Skipped 1 previous similar message [16780.774234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16780.782662] Lustre: Skipped 1 previous similar message [16785.377264] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16785.652302] Lustre: server umount lustre-MDT0000 complete [16789.100354] LustreError: 505995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943012 with bad export cookie 5426914633730808755 [16789.104814] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16789.114599] LustreError: 505995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16789.531941] Lustre: server umount lustre-MDT0001 complete [16801.175804] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [16804.044239] Key type lgssc unregistered [16804.341280] LNet: 509930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16804.350917] LNetError: 509930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16804.369710] LNet: Removed LNI 192.168.204.135@tcp [16805.439298] Key type .llcrypt unregistered [16805.442993] Key type ._llcrypt unregistered [16827.190641] Key type ._llcrypt registered [16827.194535] Key type .llcrypt registered [16827.300049] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [16828.673842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16828.738822] alg: No test for adler32 (adler32-zlib) [16829.926217] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [16830.264867] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [16831.975185] Key type lgssc registered [16833.053448] Lustre: Echo OBD driver; http://www.lustre.org/ [16840.677794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16840.715230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16842.305504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16846.864737] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [16852.455350] LustreError: 511202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16852.477113] LustreError: 511202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16855.890046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16856.443612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16860.840901] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [16868.302481] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16873.663279] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16879.943790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16880.408518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16887.176975] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [16894.838838] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16905.091184] loop0: detected capacity change from 0 to 204800000 [16905.259324] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16905.267795] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16905.279873] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16911.947804] loop0: detected capacity change from 0 to 204800000 [16912.201326] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16912.213329] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16912.229720] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16917.619174] loop0: detected capacity change from 0 to 204800000 [16923.002218] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16923.137408] Lustre: fs15246-OST0000: new disk, initializing [16923.140315] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16923.147615] Lustre: 514798:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16923.199411] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16929.592300] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [16938.811458] loop1: detected capacity change from 0 to 204800000 [16944.556337] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16944.667293] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16944.735211] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16944.854590] Lustre: fs15246-MDT0000: new disk, initializing [16945.003039] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16945.026465] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16945.120188] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16946.146702] Lustre: 515920:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000be217224 x1875842750788224/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788943185 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16949.041330] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [16955.249292] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16955.265075] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16955.303606] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16973.290648] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16973.310887] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16975.095842] Lustre: server umount fs15246-MDT0000 complete [16978.406833] LustreError: 513404:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16983.332950] Lustre: server umount fs15246-OST0000 complete [16993.764772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16993.771968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16993.780546] Lustre: Skipped 1 previous similar message [16993.797051] Lustre: Skipped 1 previous similar message [16998.887338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16998.897249] Lustre: Skipped 1 previous similar message [16999.859698] Lustre: server umount lustre-OST0000 complete [17004.001104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17004.005765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17004.010721] Lustre: Skipped 1 previous similar message [17004.020178] Lustre: Skipped 1 previous similar message [17009.123239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17009.132360] Lustre: Skipped 1 previous similar message [17017.311157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17017.462063] Lustre: server umount lustre-MDT0000 complete [17019.362123] LustreError: 511202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17019.388446] LustreError: 511202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [17020.268370] LustreError: 513046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943243 with bad export cookie 7400174965798304113 [17020.270547] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17020.277165] LustreError: 513046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [17020.569692] Lustre: server umount lustre-MDT0001 complete [17029.788295] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [17032.474075] Key type lgssc unregistered [17032.828739] LNet: 519020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17032.834097] LNetError: 519020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17032.850807] LNet: Removed LNI 192.168.204.135@tcp [17033.789401] Key type .llcrypt unregistered [17033.792983] Key type ._llcrypt unregistered [17056.487290] Key type ._llcrypt registered [17056.489551] Key type .llcrypt registered [17056.698806] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [17057.860947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17057.978141] alg: No test for adler32 (adler32-zlib) [17058.992463] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [17059.300342] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [17060.935172] Key type lgssc registered [17062.116965] Lustre: Echo OBD driver; http://www.lustre.org/ [17100.723929] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [17111.505329] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [17144.478621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17144.506177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17145.763439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17145.800915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17145.895652] Lustre: lustre-MDT0000: new disk, initializing [17145.997678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17146.013135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17150.461181] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [17161.238183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17161.456427] Lustre: 523488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17161.582887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17161.589273] Lustre: Skipped 1 previous similar message [17161.746687] Lustre: lustre-MDT0001: new disk, initializing [17161.871192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17161.940193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17161.949838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17166.797865] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [17176.552985] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17182.771034] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17190.955492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17191.320907] Lustre: lustre-OST0000: new disk, initializing [17191.329357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17191.339964] Lustre: 525460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17191.515745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17198.757451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17198.788193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17198.987728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17199.781621] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [17212.463275] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17218.060727] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17218.435656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17222.788067] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17223.045807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17229.801381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17229.803465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17229.813385] LustreError: Skipped 1 previous similar message [17229.816168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17229.880034] Lustre: Skipped 1 previous similar message [17231.382848] Lustre: server umount lustre-OST0000 complete [17239.013491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17239.027606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17239.055116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17239.059431] Lustre: Skipped 1 previous similar message [17240.034347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17240.054532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17240.058774] Lustre: Skipped 1 previous similar message [17241.869315] Lustre: server umount lustre-MDT0000 complete [17245.386203] LustreError: 523480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943468 with bad export cookie 14084797892685277785 [17245.392946] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17245.393744] LustreError: 523480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17245.670932] Lustre: server umount lustre-MDT0001 complete [17273.302246] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [17285.971520] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [17320.346470] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16989 sec ========= 04:45:41 (1788943541) [17322.161111] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:45:43 (1788943543) === [17325.483988] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:45:46 (1788943546) === [17369.818201] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [17372.561983] Key type lgssc unregistered [17372.945397] LNet: 532108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17372.956679] LNetError: 532108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17372.977738] LNet: Removed LNI 192.168.204.135@tcp [17373.913975] Key type .llcrypt unregistered [17373.921997] Key type ._llcrypt unregistered