[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 503162251 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002287] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006893] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.009133] LSM: Security Framework initializing [ 0.010039] Yama: becoming mindful. [ 0.011026] SELinux: Initializing. [ 0.012075] *** VALIDATE selinux *** [ 0.020857] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024804] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027090] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028130] *** VALIDATE tmpfs *** [ 0.030138] *** VALIDATE proc *** [ 0.031286] *** VALIDATE cgroup *** [ 0.032012] *** VALIDATE cgroup2 *** [ 0.033308] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035022] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037016] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.040907] debug: unmapping init [mem 0xffffffffb6859000-0xffffffffb6860fff] [ 0.042999] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043717] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045016] ... bit width: 48 [ 0.046016] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048016] ... max period: 00007fffffffffff [ 0.049018] ... fixed-purpose events: 3 [ 0.050015] ... event mask: 000000070000000f [ 0.052241] rcu: Hierarchical SRCU implementation. [ 0.054523] smp: Bringing up secondary CPUs ... [ 0.055600] x86: Booting SMP configuration: [ 0.056027] .... node #0, CPUs: #1 #2 #3 [ 0.059675] smp: Brought up 1 node, 4 CPUs [ 0.061018] smpboot: Max logical packages: 1 [ 0.062024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.242574] node 0 deferred pages initialised in 178ms [ 0.244162] devtmpfs: initialized [ 0.245109] x86/mm: Memory block size: 128MB [ 0.248761] gcov: version magic: 0x41383552 [ 0.251329] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.252118] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.253316] pinctrl core: initialized pinctrl subsystem [ 0.254226] [ 0.254736] ************************************************************* [ 0.255016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.256014] ** ** [ 0.257014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.258020] ** ** [ 0.259015] ** This means that this kernel is built to expose internal ** [ 0.260016] ** IOMMU data structures, which may compromise security on ** [ 0.261012] ** your system. ** [ 0.262013] ** ** [ 0.263016] ** If you see this message and you are not debugging the ** [ 0.264015] ** kernel, report this immediately to your vendor! ** [ 0.265016] ** ** [ 0.266013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267014] ************************************************************* [ 0.269000] NET: Registered protocol family 16 [ 0.270455] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.275098] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.277088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.280545] cpuidle: using governor menu [ 0.282731] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.285659] PCI: Using configuration type 1 for base access [ 0.288126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.297131] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.299020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.303058] cryptd: max_cpu_qlen set to 1000 [ 0.306269] ACPI: Added _OSI(Module Device) [ 0.308020] ACPI: Added _OSI(Processor Device) [ 0.310019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.311012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.316281] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.323393] ACPI: Interpreter enabled [ 0.325067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.327017] ACPI: Using IOAPIC for interrupt routing [ 0.328153] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.332419] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.343430] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.345043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.348022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.352091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.356537] acpiphp: Slot [2] registered [ 0.357179] acpiphp: Slot [5] registered [ 0.359218] acpiphp: Slot [6] registered [ 0.360119] acpiphp: Slot [7] registered [ 0.362145] acpiphp: Slot [8] registered [ 0.363135] acpiphp: Slot [9] registered [ 0.364137] acpiphp: Slot [10] registered [ 0.366134] acpiphp: Slot [3] registered [ 0.367127] acpiphp: Slot [4] registered [ 0.369118] acpiphp: Slot [11] registered [ 0.370093] acpiphp: Slot [12] registered [ 0.372188] acpiphp: Slot [13] registered [ 0.373105] acpiphp: Slot [14] registered [ 0.374091] acpiphp: Slot [15] registered [ 0.375095] acpiphp: Slot [16] registered [ 0.376104] acpiphp: Slot [17] registered [ 0.378069] acpiphp: Slot [18] registered [ 0.379104] acpiphp: Slot [19] registered [ 0.380112] acpiphp: Slot [20] registered [ 0.382110] acpiphp: Slot [21] registered [ 0.383166] acpiphp: Slot [22] registered [ 0.385096] acpiphp: Slot [23] registered [ 0.386103] acpiphp: Slot [24] registered [ 0.388124] acpiphp: Slot [25] registered [ 0.389099] acpiphp: Slot [26] registered [ 0.391100] acpiphp: Slot [27] registered [ 0.392118] acpiphp: Slot [28] registered [ 0.394105] acpiphp: Slot [29] registered [ 0.395103] acpiphp: Slot [30] registered [ 0.397110] acpiphp: Slot [31] registered [ 0.398096] PCI host bridge to bus 0000:00 [ 0.400022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.402023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.404024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.407024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.409018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.412029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.413161] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.416458] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.419158] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.429985] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.434540] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.437021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.439018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.442025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.445610] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.448835] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.451043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.455599] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.460015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.473020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.478019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.485063] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.492016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.497020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.521022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.536697] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.546023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.552026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.565023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.572488] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.578019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.583017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.604022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.616889] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.623022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.630025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.662028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.675333] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.682019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.688018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.701019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.711922] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.719018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.726021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.748021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.759861] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.762370] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.764327] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.765360] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.768310] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.774140] iommu: Default domain type: Passthrough [ 0.778373] SCSI subsystem initialized [ 0.780162] ACPI: bus type USB registered [ 0.781124] usbcore: registered new interface driver usbfs [ 0.784117] usbcore: registered new interface driver hub [ 0.785109] usbcore: registered new device driver usb [ 0.787197] pps_core: LinuxPPS API ver. 1 registered [ 0.789014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.793069] PTP clock support registered [ 0.795432] EDAC MC: Ver: 3.0.0 [ 0.798024] PCI: Using ACPI for IRQ routing [ 0.800258] NetLabel: Initializing [ 0.801016] NetLabel: domain hash size = 128 [ 0.803014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.805104] NetLabel: unlabeled traffic allowed by default [ 0.807386] vgaarb: loaded [ 0.809322] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.811024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.819294] clocksource: Switched to clocksource kvm-clock [ 0.945696] VFS: Disk quotas dquot_6.6.0 [ 0.947259] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.949527] *** VALIDATE ramfs *** [ 0.951224] *** VALIDATE hugetlbfs *** [ 0.952663] pnp: PnP ACPI init [ 0.954986] pnp: PnP ACPI: found 6 devices [ 0.975159] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.978280] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.980322] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.982381] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.984729] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.987210] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.989963] NET: Registered protocol family 2 [ 0.992676] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.997938] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.001427] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.006977] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.015605] TCP: Hash tables configured (established 65536 bind 65536) [ 1.018649] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.022143] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.024811] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.027603] NET: Registered protocol family 1 [ 1.030731] RPC: Registered named UNIX socket transport module. [ 1.033021] RPC: Registered udp transport module. [ 1.034609] RPC: Registered tcp transport module. [ 1.036172] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.038287] NET: Registered protocol family 44 [ 1.040264] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.042790] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.044691] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.047276] PCI: CLS 0 bytes, default 64 [ 1.048983] Unpacking initramfs... [ 2.421140] debug: unmapping init [mem 0xffff96a37cc54000-0xffff96a37ffbffff] [ 2.425063] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.426806] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.428578] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.899877] Initialise system trusted keyrings [ 2.901540] Key type blacklist registered [ 2.903408] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.912201] zbud: loaded [ 2.916358] *** VALIDATE nfs *** [ 2.917356] *** VALIDATE nfs4 *** [ 2.918536] pstore: using deflate compression [ 2.921553] Platform Keyring initialized [ 3.016431] NET: Registered protocol family 38 [ 3.017895] Key type asymmetric registered [ 3.019259] Asymmetric key parser 'x509' registered [ 3.020688] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.023257] io scheduler mq-deadline registered [ 3.024500] io scheduler kyber registered [ 3.025622] io scheduler bfq registered [ 3.027079] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.029195] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.030986] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.032919] ACPI: Power Button [PWRF] [ 3.037406] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.042351] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.050935] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.055564] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.065916] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.092374] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.121736] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.126334] Non-volatile memory driver v1.3 [ 3.127651] Linux agpgart interface v0.103 [ 3.157744] virtio_blk virtio1: [vda] 135296 512-byte logical blocks (69.3 MB/66.1 MiB) [ 3.160673] vda: detected capacity change from 0 to 69271552 [ 3.175140] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.180818] vdb: detected capacity change from 0 to 1073741824 [ 3.201657] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.203648] vdc: detected capacity change from 0 to 2621440000 [ 3.218832] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.221956] vdd: detected capacity change from 0 to 2621440000 [ 3.234707] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.237854] vde: detected capacity change from 0 to 4294967296 [ 3.252196] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.254950] vdf: detected capacity change from 0 to 4294967296 [ 3.264199] libphy: Fixed MDIO Bus: probed [ 3.278521] usbcore: registered new interface driver usbserial_generic [ 3.282367] usbserial: USB Serial support registered for generic [ 3.285646] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.291798] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.294394] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.298184] mousedev: PS/2 mouse device common for all mice [ 3.301535] rtc_cmos 00:05: RTC can wake from S4 [ 3.306934] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.309791] rtc_cmos 00:05: registered as rtc0 [ 3.310782] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.312550] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.315334] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.318849] intel_pstate: CPU model not supported [ 3.323469] hid: raw HID events driver (C) Jiri Kosina [ 3.325969] usbcore: registered new interface driver usbhid [ 3.328079] usbhid: USB HID core driver [ 3.329704] drop_monitor: Initializing network drop monitor service [ 3.332175] Initializing XFRM netlink socket [ 3.334316] NET: Registered protocol family 10 [ 3.337495] Segment Routing with IPv6 [ 3.338770] NET: Registered protocol family 17 [ 3.340775] mpls_gso: MPLS GSO support [ 3.345748] RAS: Correctable Errors collector initialized. [ 3.347904] AVX version of gcm_enc/dec engaged. [ 3.349496] AES CTR mode by8 optimization enabled [ 3.421072] sched_clock: Marking stable (3421029133, 0)->(4306494167, -885465034) [ 3.423357] registered taskstats version 1 [ 3.424581] Loading compiled-in X.509 certificates [ 3.425779] zswap: loaded using pool lzo/zbud [ 3.447093] Key type big_key registered [ 3.458552] Key type encrypted registered [ 3.460221] ima: No TPM chip found, activating TPM-bypass! [ 3.462155] ima: Allocated hash algorithm: sha1 [ 3.463801] ima: No architecture policies found [ 3.465355] evm: Initialising EVM extended attributes: [ 3.467183] evm: security.selinux [ 3.468271] evm: security.ima [ 3.469295] evm: security.capability [ 3.470266] evm: HMAC attrs: 0x1 [ 3.473215] rtc_cmos 00:05: setting system clock to 2026-05-28 12:52:16 UTC (1779972736) [ 3.479652] debug: unmapping init [mem 0xffffffffb7803000-0xffffffffb79fffff] [ 3.482799] debug: unmapping init [mem 0xffffffffb6582000-0xffffffffb6858fff] [ 3.491118] Write protecting the kernel read-only data: 28672k [ 3.494204] debug: unmapping init [mem 0xffffffffb4c03000-0xffffffffb4dfffff] [ 3.497069] debug: unmapping init [mem 0xffffffffb5514000-0xffffffffb55fffff] [ 3.533386] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.539945] systemd[1]: Detected virtualization kvm. [ 3.541369] systemd[1]: Detected architecture x86-64. [ 3.542554] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.564551] systemd[1]: No hostname configured. [ 3.567076] systemd[1]: Set hostname to . [ 3.569130] random: systemd: uninitialized urandom read (16 bytes read) [ 3.571553] systemd[1]: Initializing machine ID from random generator. [ 3.634330] random: ln: uninitialized urandom read (6 bytes read) [ 3.715714] random: systemd: uninitialized urandom read (16 bytes read) [ 3.718140] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.722609] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.727211] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.351155] device-mapper: uevent: version 1.0.3 [ 4.354158] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.075533] random: fast init done [ 5.081814] virtio_net virtio0 ens2: renamed from eth0 [ 5.127273] scsi host0: ata_piix [ 5.181616] scsi host1: ata_piix [ 5.197295] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.199840] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.849349] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.975829] random: crng init done [ 9.977346] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.350749] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.411775] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.642362] SELinux: Disabled at runtime. [ 11.704864] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.713144] systemd[1]: Detected virtualization kvm. [ 11.714798] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.170494] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.175436] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.180308] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.186268] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.188354] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.193657] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.196824] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.263134] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.654046] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.939833] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.957953] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.108293] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.116044] EDAC sbridge: Ver: 1.1.2 [ 14.860250] Key type dns_resolver registered [ 15.176396] NFS: Registering the id_resolver key type [ 15.178795] Key type id_resolver registered [ 15.180592] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg112-server login: [ 45.322898] libcfs: loading out-of-tree module taints kernel. [ 45.369977] Key type ._llcrypt registered [ 45.371919] Key type .llcrypt registered [ 45.462501] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 63.809360] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 65.543540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 65.571183] alg: No test for adler32 (adler32-zlib) [ 67.123981] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 68.161881] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 69.967686] Key type lgssc registered [ 71.668560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 90.175758] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 133.326187] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 147.245579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 147.298100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.569994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 148.599876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 148.663220] Lustre: lustre-MDT0000: new disk, initializing [ 148.735433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 148.748173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 150.018104] hrtimer: interrupt took 2822476 ns [ 152.762075] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.281550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.373731] Lustre: 6501:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 167.437418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 167.445326] Lustre: Skipped 1 previous similar message [ 167.540244] Lustre: lustre-MDT0001: new disk, initializing [ 167.686067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 167.722493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 167.725863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 172.576163] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.238600] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 189.706531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 189.910795] Lustre: lustre-OST0000: new disk, initializing [ 189.914743] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 189.916658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 189.983407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.133846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 195.161398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 195.233328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 196.610139] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.104519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.305238] Lustre: lustre-OST0001: new disk, initializing [ 215.311201] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 215.368777] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.238439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 221.251802] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 221.315663] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 222.214755] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.889960] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 243.335062] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 255.156977] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing check_logdir /tmp/testlogs/ [ 260.507535] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing yml_node [ 264.464256] Lustre: DEBUG MARKER: Client: 2.17.50.143 [ 267.172285] Lustre: DEBUG MARKER: MDS: 2.17.50.143 [ 269.764601] Lustre: DEBUG MARKER: OSS: 2.17.50.143 [ 271.585697] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu May 28 08:56:42 EDT 2026 [ 289.640616] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 291.525540] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 308.192229] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.193220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.208241] Lustre: Skipped 3 previous similar messages [ 308.220672] Lustre: Skipped 2 previous similar messages [ 313.315184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.323859] Lustre: Skipped 4 previous similar messages [ 314.394644] Lustre: server umount lustre-MDT0000 complete [ 318.384876] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779973051 with bad export cookie 10512659123568231815 [ 318.392968] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.404150] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 318.435211] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.454064] Lustre: Skipped 1 previous similar message [ 318.459210] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.471550] Lustre: Skipped 1 previous similar message [ 323.555040] LustreError: 9471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.559840] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.594402] Lustre: Skipped 1 previous similar message [ 323.599022] LustreError: 9471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 325.085908] Lustre: server umount lustre-MDT0001 complete [ 329.560821] Lustre: server umount lustre-OST0000 complete [ 333.685693] Lustre: server umount lustre-OST0001 complete [ 341.775354] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 353.574544] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 398.960768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.446989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 399.502704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 399.640489] Lustre: lustre-MDT0000: new disk, initializing [ 399.755516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.788765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 405.001607] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 416.308426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.543334] Lustre: 14986:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 416.633986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 416.638879] Lustre: Skipped 1 previous similar message [ 416.749948] Lustre: lustre-MDT0001: new disk, initializing [ 416.869890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 416.921326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 416.935092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 421.391701] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 430.355553] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 434.179408] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 442.365920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 442.919224] Lustre: lustre-OST0000: new disk, initializing [ 442.934596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 443.109346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.198948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 444.229875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 444.375270] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 450.619665] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 459.064475] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 464.523470] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 464.784553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 470.097281] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 470.328637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 475.108297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.110898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.124767] LustreError: Skipped 1 previous similar message [ 475.126616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.154825] Lustre: Skipped 1 previous similar message [ 475.182538] Lustre: Skipped 1 previous similar message [ 478.717950] Lustre: server umount lustre-OST0000 complete [ 483.297891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 483.318942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.688243] Lustre: server umount lustre-MDT0000 complete [ 490.469493] LustreError: 14997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.869014] LustreError: 14977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779973226 with bad export cookie 10512659123568233348 [ 493.873107] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 493.881768] LustreError: 14977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 494.450631] Lustre: server umount lustre-MDT0001 complete [ 499.988593] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:00:31 (1779973231) [ 503.113623] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 504.606263] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:00:36 (1779973236) [ 514.080053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 514.352694] LustreError: 18992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.365356] LustreError: 18992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 514.400574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 519.326547] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 519.673814] LustreError: 18992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.792043] LustreError: 18992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.512791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.585507] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 545.164842] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 550.253453] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 559.941897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 560.036630] LustreError: 20599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.055607] LustreError: 20599:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 567.648856] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 574.646168] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 584.677940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.682840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 584.696683] Lustre: Skipped 1 previous similar message [ 584.699318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 584.726168] Lustre: Skipped 4 previous similar messages [ 590.489565] Lustre: server umount lustre-OST0000 complete [ 596.449545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 596.466251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.485326] Lustre: Skipped 1 previous similar message [ 601.780700] Lustre: server umount lustre-MDT0000 complete [ 605.156237] LustreError: 21061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.178341] LustreError: 21061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 605.866065] LustreError: 18971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779973338 with bad export cookie 10512659123568234230 [ 605.880863] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 605.896162] LustreError: 18971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 606.273868] Lustre: server umount lustre-MDT0001 complete [ 617.251352] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 620.355404] Key type lgssc unregistered [ 620.813340] LNet: 22186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.829492] LNetError: 22186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.868564] LNet: Removed LNI 192.168.201.112@tcp [ 623.473211] Key type .llcrypt unregistered [ 623.475064] Key type ._llcrypt unregistered [ 653.308439] Key type ._llcrypt registered [ 653.314349] Key type .llcrypt registered [ 653.563850] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 655.456288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 655.532219] alg: No test for adler32 (adler32-zlib) [ 656.712545] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 657.000761] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 658.703227] Key type lgssc registered [ 659.985219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 669.740286] loop: module loaded [ 669.862262] loop0: detected capacity change from 0 to 204800000 [ 670.182886] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 670.191381] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 670.628530] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.592266] loop0: detected capacity change from 0 to 204800000 [ 677.828587] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.839667] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.845660] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 683.336210] loop0: detected capacity change from 0 to 204800000 [ 689.698593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 689.731493] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.165585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 691.234661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 691.394577] Lustre: lustre-MDT0000: new disk, initializing [ 691.523140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.547666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 695.981448] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 706.711268] loop1: detected capacity change from 0 to 204800000 [ 712.785983] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 713.047222] Lustre: lustre-OST07c6: new disk, initializing [ 713.050565] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 713.057439] Lustre: Skipped 1 previous similar message [ 713.114752] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 719.655423] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 721.012913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 721.033090] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 721.071179] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 724.906764] Lustre: 24342:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 724.911409] Lustre: 24342:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 724.917126] Lustre: 24342:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 724.922547] Lustre: 24342:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 724.931467] Lustre: 24342:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 724.937605] Lustre: 24342:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 741.103507] Lustre: 24342:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 741.118088] Lustre: 24342:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 741.128148] Lustre: 24342:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 741.147049] Lustre: 24342:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 741.157779] Lustre: 24342:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 741.168917] Lustre: 24342:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 746.228932] Lustre: 24342:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 746.240934] Lustre: 24342:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 746.248896] Lustre: 24342:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 746.252607] Lustre: 24342:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.256785] Lustre: 24342:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 746.261671] Lustre: 24342:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.270295] Lustre: 24342:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 746.279249] Lustre: 24342:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.289065] Lustre: 24342:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 746.296979] Lustre: 24342:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 746.306066] Lustre: 24342:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 746.312699] Lustre: 24342:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.254139] Lustre: 25741:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 751.258714] Lustre: 25741:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 751.263201] Lustre: 25741:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 751.272450] Lustre: 25741:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.285734] Lustre: 25741:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 751.292759] Lustre: 25741:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.297352] Lustre: 25741:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 751.301949] Lustre: 25741:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.316567] Lustre: 25741:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 751.326919] Lustre: 25741:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.336831] Lustre: 25741:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 751.341925] Lustre: 25741:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.941448] Lustre: 24342:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 755.960043] Lustre: 24342:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 755.968581] Lustre: 24342:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 755.980330] Lustre: 24342:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.986203] Lustre: 24342:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 755.993767] Lustre: 24342:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.001840] Lustre: 24342:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 756.015926] Lustre: 24342:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.025528] Lustre: 24342:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 756.030244] Lustre: 24342:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.035706] Lustre: 24342:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 756.043301] Lustre: 24342:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.440663] Lustre: 24342:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 764.449343] Lustre: 24342:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 3 previous similar messages [ 764.453421] Lustre: 24342:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 764.457889] Lustre: 24342:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 764.465719] Lustre: 24342:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 764.472715] Lustre: 24342:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 764.482436] Lustre: 24342:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 764.491150] Lustre: 24342:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 764.500266] Lustre: 24342:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 764.507016] Lustre: 24342:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 764.512136] Lustre: 24342:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 764.517395] Lustre: 24342:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 823.278140] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.278925] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 823.301148] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 826.097886] Lustre: server umount lustre-OST07c6 complete [ 837.337596] Lustre: server umount lustre-MDT0000 complete [ 857.639485] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 860.536373] Key type lgssc unregistered [ 860.777320] LNet: 27866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.788566] LNetError: 27866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.799388] LNet: Removed LNI 192.168.201.112@tcp [ 861.593740] Key type .llcrypt unregistered [ 861.596232] Key type ._llcrypt unregistered [ 880.249409] Key type ._llcrypt registered [ 880.251465] Key type .llcrypt registered [ 880.347096] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 881.363700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 881.443358] alg: No test for adler32 (adler32-zlib) [ 882.490150] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 882.765282] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 884.551783] Key type lgssc registered [ 885.811882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 895.053145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 895.068638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.776163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 902.556537] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 906.218783] LustreError: 29120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 906.235786] LustreError: 29120:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 911.328728] LustreError: 29120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.418799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.297562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 919.821151] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 926.574694] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 930.180459] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 937.799652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 938.402890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 941.553436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 945.890791] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 952.811837] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 969.427344] Lustre: server umount lustre-OST0000 complete [ 976.865914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 976.871921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.881509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 977.394884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 977.395328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 982.498510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 982.501292] Lustre: Skipped 1 previous similar message [ 984.660892] Lustre: server umount lustre-MDT0000 complete [ 987.615093] Lustre: 28429:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779973704/real 1779973704] req@000000000510481f x1866437613353216/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779973720 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 987.619411] LustreError: 29119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 987.653779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.686117] LustreError: 29119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 988.707990] Lustre: 28429:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779973705/real 1779973705] req@0000000027471617 x1866437613354240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779973721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 988.751199] LustreError: 29839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779973721 with bad export cookie 8972047346631467799 [ 988.756701] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 988.758493] LustreError: 29839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 989.061227] Lustre: server umount lustre-MDT0001 complete [ 1000.496473] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1003.336367] Key type lgssc unregistered [ 1003.652139] LNet: 32505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.659582] LNetError: 32505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.677286] LNet: Removed LNI 192.168.201.112@tcp [ 1004.598406] Key type .llcrypt unregistered [ 1004.600376] Key type ._llcrypt unregistered [ 1026.265172] Key type ._llcrypt registered [ 1026.271430] Key type .llcrypt registered [ 1026.477771] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1028.342187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1028.404716] alg: No test for adler32 (adler32-zlib) [ 1029.458989] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 1029.715606] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1031.391230] Key type lgssc registered [ 1032.485243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1040.464929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1040.479514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.174073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.959462] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1052.640249] LustreError: 33762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1052.654579] LustreError: 33762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1056.286897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1056.839252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1061.534839] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1067.691524] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1072.118049] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1080.124926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1080.849786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.026308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1089.073529] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1096.735893] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1108.455907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1108.457766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.459152] LustreError: Skipped 1 previous similar message [ 1108.464308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1108.467994] Lustre: Skipped 1 previous similar message [ 1112.921766] Lustre: server umount lustre-OST0000 complete [ 1118.179914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1118.184350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.201080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1118.203295] Lustre: Skipped 1 previous similar message [ 1123.505430] Lustre: server umount lustre-MDT0000 complete [ 1123.827452] LustreError: 34731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1123.875423] LustreError: 34731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1128.464056] LustreError: 33743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779973861 with bad export cookie 10609236726949335511 [ 1128.480985] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1128.483755] LustreError: 33743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1128.788752] Lustre: server umount lustre-MDT0001 complete [ 1140.787820] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1143.597898] Key type lgssc unregistered [ 1143.912303] LNet: 36945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.921446] LNetError: 36945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1144.999479] LNet: Removed LNI 192.168.201.112@tcp [ 1145.825714] Key type .llcrypt unregistered [ 1145.827539] Key type ._llcrypt unregistered [ 1170.048823] Key type ._llcrypt registered [ 1170.057366] Key type .llcrypt registered [ 1170.218419] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1171.060389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1171.273913] alg: No test for adler32 (adler32-zlib) [ 1172.358061] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 1172.618586] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1174.337387] Key type lgssc registered [ 1175.446850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1182.697628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1182.718303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.328188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.413760] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1194.481371] LustreError: 38345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1194.500672] LustreError: 38345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1198.448123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.803848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1203.634792] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1210.620244] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1215.187794] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1223.175161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1223.591473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.762080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1231.100592] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1237.197865] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1244.822445] Lustre: server umount lustre-MDT0000 complete [ 1245.020140] LustreError: 38339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1245.152863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1245.165982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.901094] LustreError: 38324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779973981 with bad export cookie 14542236120814518351 [ 1248.918135] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1250.074851] LustreError: 38341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.095630] LustreError: 38341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1250.145600] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [ 1250.272784] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.285859] Lustre: Skipped 2 previous similar messages [ 1253.280403] LustreError: 38341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.305677] LustreError: 38341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1253.320867] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1253.338412] Lustre: Skipped 1 previous similar message [ 1255.328443] Lustre: server umount lustre-MDT0001 complete [ 1277.558472] Lustre: server umount lustre-OST0000 complete [ 1300.784935] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1305.179925] Key type lgssc unregistered [ 1305.481442] LNet: 41869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.529618] LNetError: 41869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1305.548084] LNet: Removed LNI 192.168.201.112@tcp [ 1306.226557] Key type .llcrypt unregistered [ 1306.228946] Key type ._llcrypt unregistered [ 1334.045970] Key type ._llcrypt registered [ 1334.048261] Key type .llcrypt registered [ 1334.152410] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1335.345069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1335.487625] alg: No test for adler32 (adler32-zlib) [ 1336.576334] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 1336.767458] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1338.439382] Key type lgssc registered [ 1339.797481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1348.762154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1348.810032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.554286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.299577] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1360.867521] LustreError: 43275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.883101] LustreError: 43275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1363.629173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.894340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1368.276888] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1373.934568] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1377.921714] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1385.342627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1385.856292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.072198] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1395.066066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1396.992990] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1404.216800] Lustre: lustre-OST0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 1404.385063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1404.387821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.413567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1405.414688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1405.428305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.441526] Lustre: Skipped 2 previous similar messages [ 1408.877606] Lustre: server umount lustre-OST0000 complete [ 1430.496420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1430.508099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.551782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.022345] Lustre: server umount lustre-MDT0000 complete [ 1436.635869] LustreError: 43254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779974169 with bad export cookie 635526401267421927 [ 1436.641147] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1436.647513] LustreError: 43254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1443.071354] Lustre: server umount lustre-MDT0001 complete [ 1459.698919] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1464.401942] Key type lgssc unregistered [ 1465.028573] LNet: 46634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1465.032637] LNetError: 46634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1465.053792] LNet: Removed LNI 192.168.201.112@tcp [ 1466.516799] Key type .llcrypt unregistered [ 1466.519335] Key type ._llcrypt unregistered [ 1496.288643] Key type ._llcrypt registered [ 1496.290748] Key type .llcrypt registered [ 1496.374707] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1497.489434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1497.598184] alg: No test for adler32 (adler32-zlib) [ 1498.834680] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 1499.285774] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1501.007242] Key type lgssc registered [ 1501.996116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1510.882296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1510.902353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1512.475504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1516.629506] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1522.668234] LustreError: 48032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1522.713973] LustreError: 48032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1526.621916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.119894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1531.578045] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1539.692608] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1544.637928] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1555.968294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1556.515227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1557.575378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1564.007189] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1570.193491] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1578.978987] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1582.327762] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1594.413907] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-05-28 9h18m45s [ 1596.067632] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1598.434663] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.443845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1598.458150] Lustre: Skipped 1 previous similar message [ 1599.524152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.540934] Lustre: Skipped 4 previous similar messages [ 1602.876837] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 1604.624480] Lustre: server umount lustre-MDT0000 complete [ 1605.611339] LustreError: 48025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1605.642840] LustreError: 48025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1608.512440] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1610.004838] LustreError: 48025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.927692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.031677] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1615.309067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1615.373430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1616.381286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1620.457542] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1624.291877] Lustre: DEBUG MARKER: Wait for df (36130) ... [ 1626.545642] Lustre: DEBUG MARKER: done [ 1631.741720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.758000] Lustre: Skipped 1 previous similar message [ 1631.761383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1631.770565] Lustre: Skipped 2 previous similar messages [ 1636.835942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1636.858916] Lustre: Skipped 1 previous similar message [ 1637.366790] Lustre: server umount lustre-OST0000 complete [ 1641.962397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1641.963380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1641.988054] Lustre: Skipped 1 previous similar message [ 1647.079169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.082619] Lustre: Skipped 4 previous similar messages [ 1647.524166] Lustre: server umount lustre-MDT0000 complete [ 1651.753764] LustreError: 48014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779974384 with bad export cookie 10581127513513918458 [ 1651.756165] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1651.766631] LustreError: 48014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1652.144374] Lustre: server umount lustre-MDT0001 complete [ 1663.548956] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1666.277431] Key type lgssc unregistered [ 1666.547701] LNet: 52595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.557570] LNetError: 52595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.576778] LNet: Removed LNI 192.168.201.112@tcp [ 1667.228679] Key type .llcrypt unregistered [ 1667.230457] Key type ._llcrypt unregistered [ 1709.105469] Key type ._llcrypt registered [ 1709.107584] Key type .llcrypt registered [ 1709.189763] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1710.214453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1710.253577] alg: No test for adler32 (adler32-zlib) [ 1711.323413] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 1711.528765] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1713.199202] Key type lgssc registered [ 1714.515920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1721.597808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1721.616666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.935692] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1722.971323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1723.190767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1727.161929] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1735.878070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1735.994481] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1736.042567] Lustre: 54788:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1736.322312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1740.562592] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1746.221372] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1750.008054] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1756.877066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1757.115681] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1757.357794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.824628] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1763.894585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1768.198971] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1774.671956] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-05-28 9h21m46s [ 1776.364520] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1782.892537] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1785.340209] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1785.415541] LustreError: 55826:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1805.635310] Lustre: lustre-MDT0000: Client b6ed0968-bb9f-4037-b407-67e7046d2a29 (at 192.168.201.12@tcp) reconnecting [ 1825.511320] LustreError: 55826:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1829.079243] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1833.528093] Lustre: DEBUG MARKER: done [ 1845.733373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1845.739186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1845.752113] LustreError: Skipped 1 previous similar message [ 1845.757586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1845.813609] Lustre: Skipped 1 previous similar message [ 1849.969809] Lustre: server umount lustre-OST0000 complete [ 1855.047441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.055727] Lustre: Skipped 1 previous similar message [ 1855.058957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1855.068245] Lustre: Skipped 1 previous similar message [ 1856.097496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1856.101121] Lustre: Skipped 1 previous similar message [ 1861.848268] Lustre: server umount lustre-MDT0000 complete [ 1865.545470] LustreError: 54778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779974598 with bad export cookie 311123764966998599 [ 1865.560969] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1865.564491] LustreError: 54778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1866.058693] Lustre: server umount lustre-MDT0001 complete [ 1879.077628] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1882.971983] Key type lgssc unregistered [ 1883.248634] LNet: 58759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.257716] LNetError: 58759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.279052] LNet: Removed LNI 192.168.201.112@tcp [ 1884.528584] Key type .llcrypt unregistered [ 1884.550565] Key type ._llcrypt unregistered [ 1946.744403] Key type ._llcrypt registered [ 1946.748864] Key type .llcrypt registered [ 1946.902383] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1948.750203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1948.766341] alg: No test for adler32 (adler32-zlib) [ 1950.027576] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 1950.528594] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1952.247420] Key type lgssc registered [ 1953.664764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1957.045692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1957.071884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.523633] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1958.566498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1958.813059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1960.104521] Lustre: Failing over lustre-MDT0000 [ 1960.414517] Lustre: server umount lustre-MDT0000 complete [ 1968.929762] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:24:59 (1779974699) [ 1977.263806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.005632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.098994] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1993.771761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.986828] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1994.076196] Lustre: 61544:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1994.421193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1999.901747] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2007.723360] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2012.939467] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2020.626447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2020.749712] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2020.951235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.225460] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2028.784052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2034.015850] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2040.143838] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2049.515195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.533720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2049.539307] Lustre: Skipped 2 previous similar messages [ 2049.562505] Lustre: Skipped 2 previous similar messages [ 2054.627035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2054.635062] Lustre: Skipped 2 previous similar messages [ 2059.748253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2059.758635] Lustre: Skipped 2 previous similar messages [ 2061.279124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2061.465333] Lustre: server umount lustre-MDT0000 complete [ 2064.864675] LustreError: 61536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779974797 with bad export cookie 15236687949145632117 [ 2064.883029] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2064.890231] LustreError: 61536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2065.402772] Lustre: server umount lustre-MDT0001 complete [ 2068.456542] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2073.925609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2084.207308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.372362] Lustre: 60469:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779974803/real 1779974803] req@000000008e0ad696 x1866438733243648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779974819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2086.390659] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.803707] LustreError: 63167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.828316] LustreError: 63167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2089.924275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.069634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2090.912210] Lustre: 60469:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779974807/real 1779974807] req@000000005b1c42d3 x1866438733243904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779974823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2091.941851] LustreError: 63167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2094.653258] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2095.007082] Lustre: 60467:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779974812/real 1779974812] req@000000006a8964db x1866438733244032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779974828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2095.014425] LustreError: 63167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2095.018195] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2100.150771] Lustre: 60467:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779974817/real 1779974817] req@00000000f7279698 x1866438733244160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779974833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2101.218963] LustreError: 63166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.262556] LustreError: 63166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2104.768462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.126562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2110.229231] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2110.435131] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2119.616956] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2124.788394] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2136.034604] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.036849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2136.052212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2136.068684] Lustre: Skipped 2 previous similar messages [ 2141.166400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2141.195536] Lustre: Skipped 2 previous similar messages [ 2142.259909] Lustre: server umount lustre-MDT0000 complete [ 2146.277029] LustreError: 64992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.316574] LustreError: 64992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2146.558620] LustreError: 61537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779974879 with bad export cookie 15236687949145634392 [ 2146.560167] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2146.578469] LustreError: 61537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2147.191601] Lustre: server umount lustre-MDT0001 complete [ 2154.498103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2165.190608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.775919] Lustre: 60466:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779974884/real 1779974884] req@000000004db43056 x1866438733272832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779974900 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2167.835426] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.876245] Lustre: Skipped 2 previous similar messages [ 2170.977524] LustreError: 60465:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b69860ef x1866438733273600/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2171.220424] LustreError: 63166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.318311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.368963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2176.504878] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2176.559480] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2177.504306] Lustre: 60468:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779974894/real 1779974894] req@00000000209a5373 x1866438733273216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779974910 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2177.550609] Lustre: 60468:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2185.896235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.675311] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2191.857622] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2198.309305] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2203.711554] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2212.323440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.331707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2212.343223] Lustre: Skipped 2 previous similar messages [ 2212.371837] Lustre: Skipped 2 previous similar messages [ 2218.489571] Lustre: server umount lustre-MDT0000 complete [ 2222.554514] LustreError: 61536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779974955 with bad export cookie 15236687949145635771 [ 2222.555949] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2222.562981] LustreError: 61536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2222.593791] LustreError: 67454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.600273] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.615577] LustreError: 67454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2222.888924] Lustre: server umount lustre-MDT0001 complete [ 2231.024761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2235.229398] Lustre: DEBUG MARKER: files compared the same [ 2237.783803] Lustre: server umount lustre-OST0000 complete [ 2257.734871] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2261.191692] Key type lgssc unregistered [ 2261.590041] LNet: 70488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.604561] LNetError: 70488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2262.645502] LNet: Removed LNI 192.168.201.112@tcp [ 2263.935319] Key type .llcrypt unregistered [ 2263.937794] Key type ._llcrypt unregistered [ 2292.423574] Key type ._llcrypt registered [ 2292.425642] Key type .llcrypt registered [ 2292.605912] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2294.058300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2294.115708] alg: No test for adler32 (adler32-zlib) [ 2295.288616] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 2295.468320] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2297.103272] Key type lgssc registered [ 2298.218654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2305.620038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2305.639588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2307.207429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.976569] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug +malloc all [ 2317.810435] LustreError: 71891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.830328] LustreError: 71891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2321.403894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2321.789553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2326.625086] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug +malloc all [ 2332.887175] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2337.552917] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2344.381562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2344.653771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.089190] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug +malloc all [ 2352.892612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2357.188449] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2367.457071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2367.462301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.473099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2368.480319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2368.499545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.515478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2371.450153] Lustre: server umount lustre-OST0000 complete [ 2378.216705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2378.231948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.246596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.249176] Lustre: Skipped 1 previous similar message [ 2381.977281] Lustre: server umount lustre-MDT0000 complete [ 2383.841556] LustreError: 72601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.858674] LustreError: 72601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2386.344165] LustreError: 71872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779975119 with bad export cookie 10229104034781706957 [ 2386.349564] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2386.354653] LustreError: 71872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2386.815166] Lustre: server umount lustre-MDT0001 complete [ 2398.779102] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2400.873508] Key type lgssc unregistered [ 2401.097743] LNet: 75070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2401.103513] LNetError: 75070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2401.125854] LNet: Removed LNI 192.168.201.112@tcp [ 2401.949535] Key type .llcrypt unregistered [ 2401.951887] Key type ._llcrypt unregistered [ 2428.136926] Key type ._llcrypt registered [ 2428.139868] Key type .llcrypt registered [ 2428.253874] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2429.173914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2429.295336] alg: No test for adler32 (adler32-zlib) [ 2430.276748] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 2430.628539] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2432.319390] Key type lgssc registered [ 2433.540244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2442.357092] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2442.377133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2457.959047] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2463.135713] LustreError: 76444:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [ 2489.823424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2496.012450] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2503.003624] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2511.703773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2511.954415] Lustre: *** cfs_fail_loc=706, val=0*** [ 2512.459559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.535232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2517.483557] LustreError: 76469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.498792] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2517.504369] LustreError: 76469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2522.604400] LustreError: 76469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.658451] LustreError: 76469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.688905] LustreError: 76469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2527.326632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.644539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2532.096888] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2538.841533] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2542.963403] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2548.193050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2548.197627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.213643] LustreError: Skipped 1 previous similar message [ 2548.227083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2548.247603] Lustre: Skipped 1 previous similar message [ 2548.279131] Lustre: Skipped 1 previous similar message [ 2551.444114] Lustre: server umount lustre-OST0000 complete [ 2558.456055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2558.460722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.483408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2558.488116] Lustre: Skipped 1 previous similar message [ 2563.554202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2563.566916] Lustre: Skipped 1 previous similar message [ 2568.686822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.694748] Lustre: Skipped 1 previous similar message [ 2570.207142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2570.442650] Lustre: server umount lustre-MDT0000 complete [ 2573.792572] LustreError: 78974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.820395] LustreError: 78974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2575.580367] LustreError: 76452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779975308 with bad export cookie 4804881763104129299 [ 2575.589726] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2575.591853] LustreError: 76452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2576.063735] Lustre: server umount lustre-MDT0001 complete [ 2589.684734] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2593.037291] Key type lgssc unregistered [ 2593.349182] LNet: 79711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2593.365352] LNetError: 79711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2593.402284] LNet: Removed LNI 192.168.201.112@tcp [ 2594.337820] Key type .llcrypt unregistered [ 2594.341353] Key type ._llcrypt unregistered [ 2627.578643] Key type ._llcrypt registered [ 2627.580691] Key type .llcrypt registered [ 2627.669726] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2628.695396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2628.745453] alg: No test for adler32 (adler32-zlib) [ 2629.790859] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 2630.086867] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2631.727824] Key type lgssc registered [ 2633.228151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2642.383604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2642.406508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2648.333523] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2657.968363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.456401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2663.559109] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2663.913268] LustreError: 81781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2663.931761] LustreError: 81781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2668.007743] LustreError: 81785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.031851] LustreError: 81785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2673.122920] LustreError: 81785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.143572] LustreError: 81785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2673.167034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2673.660213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2678.753811] LustreError: 81785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.795893] LustreError: 81785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2681.201866] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2683.875898] LustreError: 81785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.920853] LustreError: 81785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2691.258990] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2691.271280] Lustre: MGS: Client 60b91d04-b185-4122-9193-a02854412d0e (at 0@lo) reconnecting [ 2691.290271] LustreError: 81796:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2691.596457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.672027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2696.565708] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2712.053199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.061382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2712.083273] Lustre: Skipped 1 previous similar message [ 2712.107375] Lustre: Skipped 1 previous similar message [ 2717.158117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2717.172899] Lustre: Skipped 1 previous similar message [ 2717.509100] Lustre: server umount lustre-OST0000 complete [ 2722.272902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2722.273842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.284050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2722.298375] Lustre: Skipped 1 previous similar message [ 2727.069063] Lustre: server umount lustre-MDT0000 complete [ 2727.397386] LustreError: 81786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.411163] LustreError: 81786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2731.152533] Lustre: server umount lustre-MDT0001 complete [ 2735.131904] Lustre: server umount MGS complete [ 2748.263769] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:37:59 (1779975479) [ 2779.419901] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 2789.068050] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2826.144275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.749437] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2839.778772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.026636] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2840.072473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2840.136721] Lustre: lustre-MDT0001: new disk, initializing [ 2840.197075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2840.226322] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2840.234495] Lustre: 88866:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2840.245831] LustreError: 88866:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2844.707365] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2855.841448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2856.166620] Lustre: lustre-OST0000: new disk, initializing [ 2856.171688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2856.275466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.653459] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2863.272671] Lustre: 89906:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000005a97d514 x1866439446274176/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1779975612 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2870.217076] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2877.577428] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.587470] Lustre: MGS: Client e2f245bc-1e36-41e0-9462-3c503d475062 (at 0@lo) reconnecting [ 2877.602710] LustreError: 88852:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2877.629149] Lustre: 90549:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2877.662576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2877.759861] Lustre: lustre-MDT0000: new disk, initializing [ 2878.002461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.042533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2878.078341] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2879.138483] Lustre: 90575:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000047f10ea8 x1866439446280576/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1779975628 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2882.426775] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2884.259701] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2888.229807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2888.238461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2888.328459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2913.255365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.259955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2913.274216] Lustre: Skipped 1 previous similar message [ 2913.287819] Lustre: Skipped 2 previous similar messages [ 2918.373347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2918.382694] Lustre: Skipped 1 previous similar message [ 2918.786966] Lustre: server umount lustre-OST0000 complete [ 2923.489113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2923.504099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2923.526433] Lustre: Skipped 1 previous similar message [ 2928.609284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2928.610994] Lustre: Skipped 2 previous similar messages [ 2928.897473] Lustre: server umount lustre-MDT0000 complete [ 2933.728958] LustreError: 90986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.760667] LustreError: 90986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2940.709428] Lustre: server umount lustre-MDT0001 complete [ 2945.155818] Lustre: server umount MGS complete [ 2961.530765] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:41:31 (1779975691) [ 2980.538736] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2983.595709] Key type lgssc unregistered [ 2983.804768] LNet: 93043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2983.810370] LNetError: 93043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2984.874201] LNet: Removed LNI 192.168.201.112@tcp [ 2985.657903] Key type .llcrypt unregistered [ 2985.659641] Key type ._llcrypt unregistered [ 3005.647579] Key type ._llcrypt registered [ 3005.649863] Key type .llcrypt registered [ 3005.738185] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3007.201820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3007.242180] alg: No test for adler32 (adler32-zlib) [ 3008.222716] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 3008.449445] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3010.167437] Key type lgssc registered [ 3011.529332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3019.463675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3019.510697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.981357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3026.382291] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3030.498411] LustreError: 94249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.517891] LustreError: 94249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3034.831685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3035.167542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3039.383418] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3045.797183] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3051.763204] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3060.460233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3060.711886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.854303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3067.020053] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3072.395895] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3081.187160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3081.189932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3081.213669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3086.306995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3086.311995] Lustre: Skipped 2 previous similar messages [ 3086.763203] Lustre: server umount lustre-OST0000 complete [ 3091.424663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.427750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3091.437149] Lustre: Skipped 2 previous similar messages [ 3091.437496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3091.466421] Lustre: Skipped 1 previous similar message [ 3096.549435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3096.562350] Lustre: Skipped 1 previous similar message [ 3101.664387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3101.671815] Lustre: Skipped 1 previous similar message [ 3105.247160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3105.537774] Lustre: server umount lustre-MDT0000 complete [ 3106.787192] LustreError: 94249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3106.834871] LustreError: 94249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3110.934061] LustreError: 94230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779975843 with bad export cookie 2110560935362343754 [ 3110.944324] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3110.955412] LustreError: 94230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3111.441490] Lustre: server umount lustre-MDT0001 complete [ 3124.159927] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3126.545189] Key type lgssc unregistered [ 3126.822565] LNet: 97434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3126.831669] LNetError: 97434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3126.853734] LNet: Removed LNI 192.168.201.112@tcp [ 3127.741737] Key type .llcrypt unregistered [ 3127.744960] Key type ._llcrypt unregistered [ 3144.824697] Key type ._llcrypt registered [ 3144.826366] Key type .llcrypt registered [ 3144.962039] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3146.311795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3146.347787] alg: No test for adler32 (adler32-zlib) [ 3147.444330] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 3147.676728] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3149.351192] Key type lgssc registered [ 3150.311305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3156.790941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3156.795807] LustreError: 98503:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 3156.811491] LustreError: 98505:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 3156.817673] LustreError: 98503:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 3156.823290] LustreError: 98503:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3156.832739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.834361] LustreError: 98503:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3158.649278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.968890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3168.185640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3172.532269] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3178.306177] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing lsmod [ 3181.751118] LustreError: 99666:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 3181.837499] LustreError: 99676:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 3181.844807] LustreError: 99666:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4912 [ 3181.848804] LustreError: 99666:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3181.862283] LustreError: 99666:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3181.865464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3182.331159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.379647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3188.707327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3188.720935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.743119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3189.736983] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.738643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3189.761587] Lustre: Skipped 1 previous similar message [ 3189.766365] Lustre: Skipped 1 previous similar message [ 3191.700879] Lustre: server umount lustre-MDT0000 complete [ 3193.907754] LustreError: 99041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3193.938420] LustreError: 99041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3195.257464] LustreError: 98517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779975928 with bad export cookie 16994142480390529393 [ 3195.262242] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3195.270186] LustreError: 98517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3196.000271] LustreError: 98972:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000099918108 x1866439988368512/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3196.165569] Lustre: server umount lustre-MDT0001 complete [ 3210.371563] Lustre: server umount lustre-OST0000 complete [ 3220.296346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.764794] LustreError: 100742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3220.908669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.616647] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3226.081513] LustreError: 100742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.180319] LustreError: 100742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.308923] LustreError: 100742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3236.518380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.777168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3241.572843] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3248.423902] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3253.078806] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3260.582498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3260.871812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.007443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3267.270640] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3274.006929] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3282.403108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3282.423465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.442390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3283.005655] Lustre: server umount lustre-OST0000 complete [ 3288.032568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3288.049148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.076423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3295.712913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3295.727321] Lustre: Skipped 3 previous similar messages [ 3297.902288] Lustre: server umount lustre-MDT0000 complete [ 3300.835584] LustreError: 102801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3300.880737] LustreError: 102801:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3304.364029] LustreError: 100721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779976037 with bad export cookie 16994142480390530324 [ 3304.373492] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3304.375342] LustreError: 100721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3304.628114] Lustre: server umount lustre-MDT0001 complete [ 3317.656870] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3320.969513] Key type lgssc unregistered [ 3321.414087] LNet: 103925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3321.421273] LNetError: 103925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3321.445372] LNet: Removed LNI 192.168.201.112@tcp [ 3322.272511] Key type .llcrypt unregistered [ 3322.275770] Key type ._llcrypt unregistered [ 3348.042842] Key type ._llcrypt registered [ 3348.044985] Key type .llcrypt registered [ 3348.129728] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3349.214881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3349.241036] alg: No test for adler32 (adler32-zlib) [ 3350.260213] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 3350.496060] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3352.167184] Key type lgssc registered [ 3353.475217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3361.625698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3361.655041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3363.192794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.008983] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3373.562871] LustreError: 105326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3373.603734] LustreError: 105326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3377.788459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.530603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3384.383816] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3390.370603] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3394.508614] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3402.538696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3402.974872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3408.203843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3408.906397] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3415.425173] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3421.739284] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3424.769984] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3429.861020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3429.870183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.880912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3433.954668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3433.974090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3433.993404] Lustre: Skipped 1 previous similar message [ 3435.570468] Lustre: server umount lustre-OST0000 complete [ 3440.097990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3440.104829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.131088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3444.194135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3444.195762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3444.233150] Lustre: Skipped 1 previous similar message [ 3445.942881] Lustre: server umount lustre-MDT0000 complete [ 3449.312354] LustreError: 106235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.326661] LustreError: 106235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3449.417441] LustreError: 105307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779976182 with bad export cookie 3632090131738965928 [ 3449.419134] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3449.427687] LustreError: 105307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3449.811269] Lustre: server umount lustre-MDT0001 complete [ 3462.474818] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3467.247653] Key type lgssc unregistered [ 3467.657891] LNet: 108613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3467.679376] LNetError: 108613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3467.702317] LNet: Removed LNI 192.168.201.112@tcp [ 3468.765073] Key type .llcrypt unregistered [ 3468.766758] Key type ._llcrypt unregistered [ 3488.666220] Key type ._llcrypt registered [ 3488.667983] Key type .llcrypt registered [ 3488.760403] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3490.066017] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3490.154205] alg: No test for adler32 (adler32-zlib) [ 3491.477803] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 3491.738937] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3493.447395] Key type lgssc registered [ 3494.739725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3504.297905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3504.311514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.143556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3511.792648] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3516.388444] LustreError: 109820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3516.429363] LustreError: 109820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3521.518592] LustreError: 109820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.358991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.936742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3528.685620] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3535.076379] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3540.117928] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3547.212706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3547.466249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3551.610329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3553.063906] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3559.060829] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3569.120232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3569.129431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3569.150663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3572.200457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3572.207656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3572.233309] Lustre: Skipped 1 previous similar message [ 3573.326460] Lustre: server umount lustre-OST0000 complete [ 3577.336468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3577.356503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3582.433851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3582.437920] Lustre: Skipped 2 previous similar messages [ 3587.552479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3591.135154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3591.322741] Lustre: server umount lustre-MDT0000 complete [ 3592.674192] LustreError: 109816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3592.689454] LustreError: 109816:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3595.577045] LustreError: 109801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779976328 with bad export cookie 14587525450911357148 [ 3595.605461] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3596.225136] Lustre: server umount lustre-MDT0001 complete [ 3609.381421] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3611.973040] Key type lgssc unregistered [ 3612.290635] LNet: 113000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3612.318191] LNetError: 113000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3613.351509] LNet: Removed LNI 192.168.201.112@tcp [ 3614.270656] Key type .llcrypt unregistered [ 3614.273445] Key type ._llcrypt unregistered [ 3638.476433] Key type ._llcrypt registered [ 3638.479485] Key type .llcrypt registered [ 3638.606865] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3639.688086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3639.788900] alg: No test for adler32 (adler32-zlib) [ 3640.831596] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 3641.096929] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3642.863183] Key type lgssc registered [ 3644.429882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3653.181715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3653.195571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.102678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3662.319540] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3665.383451] LustreError: 114401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3665.403331] LustreError: 114401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3670.505501] LustreError: 114401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3671.822660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3672.374965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3677.078293] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3682.449917] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3686.247491] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3692.904768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3693.296327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3700.522715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3701.175144] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3707.020304] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3714.260389] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3718.675603] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3718.683739] Lustre: Skipped 2 previous similar messages [ 3721.875582] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3721.926717] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3723.912761] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3723.926666] Lustre: Skipped 2 previous similar messages [ 3730.036479] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3730.044735] Lustre: Skipped 1 previous similar message [ 3735.656535] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.12@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3735.661327] Lustre: Skipped 2 previous similar messages [ 3735.993563] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.12@tcp 192.168.0.[2,10]@tcp [ 3751.370778] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.12/24@tcp in log lustre-MDT0000 [ 3751.380283] Lustre: Skipped 2 previous similar messages [ 3759.240562] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.12/24@tcp [ 3759.251804] Lustre: Skipped 1 previous similar message [ 3767.267065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3767.278506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3767.285687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3767.303638] Lustre: Skipped 1 previous similar message [ 3767.311129] Lustre: Skipped 1 previous similar message [ 3767.536540] Lustre: server umount lustre-OST0000 complete [ 3772.396049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.398842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3772.417992] Lustre: Skipped 1 previous similar message [ 3777.504652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3777.513717] Lustre: Skipped 2 previous similar messages [ 3779.813454] Lustre: server umount lustre-MDT0000 complete [ 3782.627729] LustreError: 114401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3782.669285] LustreError: 114401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3783.617825] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3783.620359] LustreError: 114382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779976516 with bad export cookie 15901991544107161024 [ 3783.639708] LustreError: 114382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3783.926929] Lustre: server umount lustre-MDT0001 complete [ 3794.291287] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3797.031384] Key type lgssc unregistered [ 3797.432318] LNet: 118972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3797.442160] LNetError: 118972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3797.454613] LNet: Removed LNI 192.168.201.112@tcp [ 3798.437556] Key type .llcrypt unregistered [ 3798.440909] Key type ._llcrypt unregistered [ 3825.204460] Key type ._llcrypt registered [ 3825.207134] Key type .llcrypt registered [ 3825.308675] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3826.220523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3826.318680] alg: No test for adler32 (adler32-zlib) [ 3827.383304] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 3827.607868] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3829.256759] Key type lgssc registered [ 3830.273225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3836.591567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3836.610259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.983878] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3838.005686] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.12/24@tcp [ 3838.049125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.370126] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3843.042024] LustreError: 120517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3843.062303] LustreError: 120517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3847.143121] LustreError: 120517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.715231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3850.196922] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3850.237602] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.12/24@tcp [ 3850.273103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3854.833303] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3860.663752] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3865.226625] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3872.350621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3872.603273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3878.363649] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3882.944271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3883.825753] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3894.764798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3894.777514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.791283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3896.320489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3896.330828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3896.358162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3898.361956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3898.369035] Lustre: Skipped 1 previous similar message [ 3900.558731] Lustre: server umount lustre-OST0000 complete [ 3906.528637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3906.539561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3906.556495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3908.577382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3910.947948] Lustre: server umount lustre-MDT0000 complete [ 3913.702702] LustreError: 120517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3913.709464] LustreError: 120517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3914.781877] LustreError: 120496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779976647 with bad export cookie 6883889545864524256 [ 3914.798386] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3915.173414] Lustre: server umount lustre-MDT0001 complete [ 3925.211993] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3928.097803] Key type lgssc unregistered [ 3928.344196] LNet: 123750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3928.351933] LNetError: 123750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3928.366803] LNet: Removed LNI 192.168.201.112@tcp [ 3929.220329] Key type .llcrypt unregistered [ 3929.224879] Key type ._llcrypt unregistered [ 3955.939744] Key type ._llcrypt registered [ 3955.945384] Key type .llcrypt registered [ 3956.030850] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 3970.387395] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3971.377353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3971.403946] alg: No test for adler32 (adler32-zlib) [ 3972.390921] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 3972.580311] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3974.231235] Key type lgssc registered [ 3975.378277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4016.838538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4016.875902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4018.252131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4018.295850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4018.482326] Lustre: lustre-MDT0000: new disk, initializing [ 4018.646706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4018.686751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4022.690733] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4034.002483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4034.100438] Lustre: 127118:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4034.143967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4034.146836] Lustre: Skipped 1 previous similar message [ 4034.242183] Lustre: lustre-MDT0001: new disk, initializing [ 4034.359920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4034.411364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4034.429403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4039.361213] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4049.041768] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4053.935476] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4062.366953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4062.807154] Lustre: lustre-OST0000: new disk, initializing [ 4062.818334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4062.983089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.437116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4068.450577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4068.571402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4069.648327] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4078.509606] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4083.498693] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4083.712676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4087.620251] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4087.893945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4094.433049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4094.434115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4094.460619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4094.472743] Lustre: Skipped 1 previous similar message [ 4099.560024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4099.569848] Lustre: Skipped 1 previous similar message [ 4104.159149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4104.330138] Lustre: server umount lustre-OST0000 complete [ 4109.806403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.817807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.825654] Lustre: Skipped 2 previous similar messages [ 4114.560373] Lustre: server umount lustre-MDT0000 complete [ 4119.894472] LustreError: 127112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779976852 with bad export cookie 16754264162183693266 [ 4119.900673] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4119.913608] LustreError: 127112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4120.512423] Lustre: server umount lustre-MDT0001 complete [ 4130.362889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4130.677156] LustreError: 130945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4130.694097] LustreError: 130945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4130.737443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4134.909438] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4135.916499] LustreError: 130945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4140.006410] LustreError: 130945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.023600] LustreError: 130945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4144.620874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4144.937751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4149.593982] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4155.903299] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4159.910398] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4167.270041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4167.641037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4174.368781] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4180.481539] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4185.525250] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4185.883766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4191.709718] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4192.111110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4202.836747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4203.061471] Lustre: lustre-OST0001: new disk, initializing [ 4203.065375] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4203.158690] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4209.280028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4209.294239] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4209.451967] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4210.155943] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4218.505679] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4218.774155] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4223.759425] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4224.060805] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4228.759779] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cd306d29800.ost_server_uuid 50 [ 4230.470770] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8cd306d29800.ost_server_uuid in FULL state after 0 sec [ 4242.740972] Lustre: server umount lustre-OST0001 complete [ 4245.482527] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4245.486603] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4245.507731] LustreError: 132540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4245.577389] LustreError: Skipped 1 previous similar message [ 4247.526176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4247.529866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4247.541212] Lustre: Skipped 1 previous similar message [ 4247.545532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4247.558261] Lustre: Skipped 1 previous similar message [ 4252.745848] Lustre: server umount lustre-OST0000 complete [ 4257.765777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4257.777803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.786161] Lustre: Skipped 1 previous similar message [ 4257.790257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4257.796124] Lustre: Skipped 2 previous similar messages [ 4263.768417] Lustre: server umount lustre-MDT0000 complete [ 4265.958911] LustreError: 132453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4265.994869] LustreError: 132453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4268.333539] LustreError: 130925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779977001 with bad export cookie 16754264162183694148 [ 4268.351759] LustreError: 130925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4268.354515] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4268.865226] Lustre: server umount lustre-MDT0001 complete [ 4290.977388] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 4293.580155] Key type lgssc unregistered [ 4293.836347] LNet: 136783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4293.842335] LNetError: 136783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4293.868547] LNet: Removed LNI 192.168.201.112@tcp [ 4294.700595] Key type .llcrypt unregistered [ 4294.702642] Key type ._llcrypt unregistered [ 4338.884971] Key type ._llcrypt registered [ 4338.887555] Key type .llcrypt registered [ 4338.999542] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 4355.581752] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4356.400338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4356.461195] alg: No test for adler32 (adler32-zlib) [ 4357.502191] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 4358.045961] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 4359.759234] Key type lgssc registered [ 4360.762115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4403.144992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4403.191711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.458647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4404.495804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4404.562242] Lustre: lustre-MDT0000: new disk, initializing [ 4404.664590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4404.687585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4409.506735] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4421.163563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.316507] Lustre: 140906:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4421.412235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4421.415334] Lustre: Skipped 1 previous similar message [ 4421.480570] Lustre: lustre-MDT0001: new disk, initializing [ 4421.564347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4421.596979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4421.606860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4426.237560] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4434.973598] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4439.441959] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4446.971971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4447.336510] Lustre: lustre-OST0000: new disk, initializing [ 4447.340802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4447.413225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4453.495991] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4457.042823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4457.068969] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4457.185535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4463.032728] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4468.894890] Lustre: Failing over lustre-OST0000 [ 4469.013427] Lustre: server umount lustre-OST0000 complete [ 4485.151149] Lustre: 139131:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779977202/real 1779977202] req@000000005ac78645 x1866441258205056/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1779977218 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4485.178086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.293538] Lustre: 139129:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779977205/real 1779977205] req@000000006099f161 x1866441258205184/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779977221 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4488.355847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.443453] Lustre: 139130:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779977205/real 1779977205] req@000000002cfbb930 x1866441258205952/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779977221 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4489.473081] Lustre: 139130:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4493.063033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4493.729267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.760531] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4494.495387] Lustre: 139131:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779977210/real 1779977210] req@000000001c759b7e x1866441258207104/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779977226 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4494.546107] Lustre: 139131:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4495.728936] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4498.877499] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4498.877525] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4499.945087] Lustre: 139130:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779977216/real 1779977216] req@000000004b3f5031 x1866441258208128/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779977232 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4501.419510] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4505.888453] Lustre: Failing over lustre-MDT0000 [ 4506.082039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4506.114052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4506.126505] Lustre: Skipped 1 previous similar message [ 4506.146126] LustreError: 141952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4506.185797] LustreError: 141952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4506.251750] Lustre: server umount lustre-MDT0000 complete [ 4511.205369] LustreError: 142363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4511.224364] LustreError: 142363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4516.325975] LustreError: 141952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.338523] LustreError: 141952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4521.440884] LustreError: 141952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4521.470764] LustreError: 141952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4526.569515] LustreError: 140913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4526.593109] LustreError: 140913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4527.131596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4527.383892] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4527.922341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4527.993549] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4532.246986] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4533.021027] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4533.237493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4533.245428] Lustre: Skipped 1 previous similar message [ 4533.272557] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4533.317442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4543.475076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.504688] Lustre: Skipped 2 previous similar messages [ 4543.521544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4543.534845] Lustre: Skipped 1 previous similar message [ 4548.586277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4548.601043] Lustre: Skipped 1 previous similar message [ 4553.702304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4553.715392] Lustre: Skipped 1 previous similar message [ 4555.231475] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4555.517436] Lustre: server umount lustre-OST0000 complete [ 4563.938720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.939826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4563.956452] Lustre: Skipped 1 previous similar message [ 4563.962095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4563.981014] Lustre: Skipped 1 previous similar message [ 4566.406077] Lustre: server umount lustre-MDT0000 complete [ 4569.059163] LustreError: 141952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4569.079735] LustreError: 141952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4571.189582] LustreError: 140899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779977304 with bad export cookie 7276309416937054290 [ 4571.202114] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4571.212289] LustreError: 140899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4571.621848] Lustre: server umount lustre-MDT0001 complete [ 4585.135434] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 4588.728264] Key type lgssc unregistered [ 4589.152180] LNet: 146153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4589.160811] LNetError: 146153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4590.215949] LNet: Removed LNI 192.168.201.112@tcp [ 4591.396180] Key type .llcrypt unregistered [ 4591.405532] Key type ._llcrypt unregistered [ 4632.675745] Key type ._llcrypt registered [ 4632.678293] Key type .llcrypt registered [ 4632.771620] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4633.987386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4634.080487] alg: No test for adler32 (adler32-zlib) [ 4635.451284] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 4635.678144] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 4637.383879] Key type lgssc registered [ 4638.661177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4647.594876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4647.656038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4649.425045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.512562] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4659.684268] LustreError: 147659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4659.708946] LustreError: 147659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4663.665344] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4664.801333] LustreError: 147659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4668.454972] LustreError: 147688:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4668.472332] LustreError: 147688:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4668.495496] LustreError: 147688:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4668.990595] Lustre: server umount lustre-MDT0000 complete [ 4686.141197] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 4697.682865] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4733.248145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4733.596226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4733.624065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4733.714054] Lustre: lustre-MDT0000: new disk, initializing [ 4733.819128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4733.842298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4738.571033] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4749.822501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4749.926657] Lustre: 150843:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4750.032860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4750.040695] Lustre: Skipped 1 previous similar message [ 4750.122816] Lustre: lustre-MDT0001: new disk, initializing [ 4750.250758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4750.279667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4750.309382] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4754.552352] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4762.363369] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4765.793904] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4772.849419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4773.242753] Lustre: lustre-OST0000: new disk, initializing [ 4773.261835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4773.382349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4781.870481] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4783.707050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4783.722193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4783.954864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4792.076829] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4798.206607] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4800.110637] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:12:11 (1779977531) [ 5093.348258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5093.360671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5093.374359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5096.417882] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.419274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5096.454450] Lustre: Skipped 2 previous similar messages [ 5098.140739] Lustre: server umount lustre-MDT0000 complete [ 5101.541361] LustreError: 153665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.563108] LustreError: 153665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5102.476608] LustreError: 150835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779977835 with bad export cookie 15300030212743286361 [ 5102.489077] LustreError: 150835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5102.496570] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5106.655683] LustreError: 150851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5106.655709] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5106.692159] Lustre: Skipped 1 previous similar message [ 5106.700850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5108.849989] Lustre: server umount lustre-MDT0001 complete [ 5119.291174] Lustre: server umount lustre-OST0000 complete [ 5129.732453] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 5146.198879] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5192.286648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5192.603144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5192.655163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5192.773318] Lustre: lustre-MDT0000: new disk, initializing [ 5192.857712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5192.891947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5197.437764] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5210.428728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5210.605348] Lustre: 157138:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5210.793351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5210.797492] Lustre: Skipped 1 previous similar message [ 5210.923427] Lustre: lustre-MDT0001: new disk, initializing [ 5211.052211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5211.117243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5211.142922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5215.662827] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5223.623927] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5228.223221] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5235.918717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5236.208870] Lustre: lustre-OST0000: new disk, initializing [ 5236.213349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5236.316384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5242.860940] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5243.506818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5243.512918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5243.660909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5250.566807] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5255.640431] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5255.904963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5261.095525] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5261.288090] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5264.353812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5264.359175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5264.361782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5264.361791] Lustre: Skipped 1 previous similar message [ 5264.397905] Lustre: Skipped 1 previous similar message [ 5269.491540] LustreError: 158865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5269.506465] LustreError: 158865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5269.684502] Lustre: server umount lustre-OST0000 complete [ 5274.595498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.627673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5274.630271] Lustre: Skipped 1 previous similar message [ 5280.345371] Lustre: server umount lustre-MDT0000 complete [ 5284.429271] LustreError: 157916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779978017 with bad export cookie 15300030212743445191 [ 5284.432081] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5284.443319] LustreError: 157916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5284.880259] Lustre: server umount lustre-MDT0001 complete [ 5294.606636] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:20:26 (1779978026) [ 5316.216543] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 5331.891880] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5375.019821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5375.361492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5375.422669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5375.565478] Lustre: lustre-MDT0000: new disk, initializing [ 5375.665601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5375.730340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5381.139446] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5393.366745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.549617] Lustre: 163477:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5393.849086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5399.101299] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5408.491556] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5413.294738] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5422.893967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5423.168259] Lustre: lustre-OST0000: new disk, initializing [ 5423.171096] Lustre: Skipped 1 previous similar message [ 5423.180183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5423.190442] Lustre: Skipped 2 previous similar messages [ 5430.498667] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5432.367193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5432.379837] Lustre: Skipped 1 previous similar message [ 5432.395187] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5432.513117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5438.387927] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5448.170771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5448.173646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5448.184979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5448.184988] Lustre: Skipped 2 previous similar messages [ 5448.236645] Lustre: Skipped 2 previous similar messages [ 5452.642536] Lustre: server umount lustre-OST0000 complete [ 5463.469361] Lustre: server umount lustre-MDT0000 complete [ 5463.520235] LustreError: 163484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5463.542235] LustreError: 163484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5467.822979] LustreError: 163470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779978200 with bad export cookie 15300030212743446073 [ 5467.825148] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5467.833113] LustreError: 163470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5468.388592] Lustre: server umount lustre-MDT0001 complete [ 5477.384734] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:23:28 (1779978208) [ 5492.984389] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 5503.295883] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5542.533030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.848913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5542.875943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5542.988643] Lustre: lustre-MDT0000: new disk, initializing [ 5543.111927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.115680] Lustre: Skipped 2 previous similar messages [ 5543.130251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5547.912553] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5560.387144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5560.513348] Lustre: 169408:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5560.965214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5565.912506] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5573.817497] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5577.552814] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5584.418213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5591.170319] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5592.806930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5598.684369] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5608.422028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5608.429338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5608.444459] Lustre: Skipped 2 previous similar messages [ 5608.449040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5608.455620] LustreError: Skipped 1 previous similar message [ 5608.477623] Lustre: Skipped 4 previous similar messages [ 5612.729897] Lustre: server umount lustre-OST0000 complete [ 5617.135423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5622.889298] Lustre: server umount lustre-MDT0000 complete [ 5623.778535] LustreError: 169415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5623.805087] LustreError: 169415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5626.987527] LustreError: 169400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779978360 with bad export cookie 15300030212743447158 [ 5626.990098] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5627.001130] LustreError: 169400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5627.592090] Lustre: server umount lustre-MDT0001 complete [ 5640.936579] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5645.902958] Key type lgssc unregistered [ 5646.468993] LNet: 172884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5646.483890] LNetError: 172884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5646.522800] LNet: Removed LNI 192.168.201.112@tcp [ 5647.964852] Key type .llcrypt unregistered [ 5647.971369] Key type ._llcrypt unregistered [ 5679.884911] Key type ._llcrypt registered [ 5679.887991] Key type .llcrypt registered [ 5679.996832] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5681.201654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5681.392591] alg: No test for adler32 (adler32-zlib) [ 5682.653237] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 5683.118423] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5684.831452] Key type lgssc registered [ 5686.263652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5693.980843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5694.002113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5695.682069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5701.376541] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5705.207964] LustreError: 174284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5705.245548] LustreError: 174284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5710.309854] LustreError: 174284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5712.890530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5713.218938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5718.182758] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5724.955354] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5729.553041] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5736.512264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5736.813848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5740.920533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5742.566577] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5748.801511] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5769.707441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5769.723232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.767889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5771.748142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5771.748372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5771.751291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.758534] Lustre: Skipped 1 previous similar message [ 5773.028071] Lustre: server umount lustre-OST0000 complete [ 5779.943173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5779.956073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.967462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5781.986830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5781.988895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5782.018049] Lustre: Skipped 1 previous similar message [ 5783.502828] Lustre: server umount lustre-MDT0000 complete [ 5787.111890] LustreError: 175009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.151928] LustreError: 175009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5788.200868] LustreError: 175003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779978521 with bad export cookie 1460672480537445342 [ 5788.202498] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5788.209061] LustreError: 175003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5788.803647] Lustre: server umount lustre-MDT0001 complete [ 5804.202634] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5808.557352] Key type lgssc unregistered [ 5809.101148] LNet: 177494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5809.118374] LNetError: 177494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5809.152039] LNet: Removed LNI 192.168.201.112@tcp [ 5810.491628] Key type .llcrypt unregistered [ 5810.494493] Key type ._llcrypt unregistered [ 5842.325439] Key type ._llcrypt registered [ 5842.332407] Key type .llcrypt registered [ 5842.551462] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5843.778933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5844.084787] alg: No test for adler32 (adler32-zlib) [ 5845.248646] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 5845.477724] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5847.175491] Key type lgssc registered [ 5848.447047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5857.190762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5857.214597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5858.607375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5862.758813] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5868.078917] LustreError: 178915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.115534] LustreError: 178915:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5873.132088] LustreError: 178915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5874.623179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5875.340784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5880.746228] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5889.576185] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5897.616453] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5907.926538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5908.344449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5917.001461] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5923.799486] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5928.824674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5936.096368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5936.107103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.142094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5939.180770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5939.195909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5939.205444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5939.246142] Lustre: Skipped 1 previous similar message [ 5941.573304] Lustre: server umount lustre-OST0000 complete [ 5950.763802] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8cd306d2a000.ost_server_uuid 50 [ 5953.456677] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5969.889924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5969.893956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5969.913285] Lustre: Skipped 1 previous similar message [ 5969.926501] Lustre: Skipped 2 previous similar messages [ 5975.009109] LustreError: 180075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5975.030739] LustreError: 180075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5976.946740] Lustre: server umount lustre-MDT0000 complete [ 5980.136538] LustreError: 180132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5980.169904] LustreError: 180132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5981.390175] LustreError: 181533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779978714 with bad export cookie 16797612947552738956 [ 5981.393880] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5981.405097] LustreError: 181533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5982.186739] Lustre: server umount lustre-MDT0001 complete [ 5992.884668] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:32:04 (1779978724) [ 6000.456551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6001.026698] LustreError: 182405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.081867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6006.905801] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6011.381556] LustreError: 182404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6011.417446] LustreError: 182404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6021.734198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6022.303744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6027.940764] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6035.900749] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6040.921693] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6050.886779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6051.353948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6059.391904] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6068.046256] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6071.874656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6077.263140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6077.521642] Lustre: lustre-OST0001: new disk, initializing [ 6077.525901] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6077.665842] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6085.773166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6085.789837] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6085.972474] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6086.169689] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6095.822523] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6101.482280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6101.502463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6106.042272] Lustre: server umount lustre-OST0000 complete [ 6106.597518] LustreError: 184004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6106.644886] LustreError: 184004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6112.943620] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6113.169672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6118.323435] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6118.696501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6137.312961] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6137.319288] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6137.326485] Lustre: Skipped 1 previous similar message [ 6137.335795] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6137.340909] Lustre: Skipped 3 previous similar messages [ 6140.181357] Lustre: server umount lustre-OST0001 complete [ 6144.993102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6145.008865] LustreError: Skipped 1 previous similar message [ 6147.553429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6147.554410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.575430] Lustre: Skipped 1 previous similar message [ 6147.577947] Lustre: Skipped 2 previous similar messages [ 6151.247957] Lustre: server umount lustre-MDT0000 complete [ 6152.674270] LustreError: 183117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6152.698823] LustreError: 183117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 6156.039176] LustreError: 184912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779978889 with bad export cookie 16797612947552740027 [ 6156.042919] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6156.057266] LustreError: 184912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6156.606990] Lustre: server umount lustre-MDT0001 complete [ 6190.250643] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:35:20 (1779978920) [ 6199.638701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6200.333372] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6200.415738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6200.945110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6206.770464] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6218.813224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6218.939793] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6218.995658] Lustre: 188206:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6224.222416] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6231.379340] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6235.584927] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6244.332820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6244.547759] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6253.865441] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6254.324920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6262.804758] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6271.466888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6271.543550] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6271.820565] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6271.822932] Lustre: Skipped 2 previous similar messages [ 6280.138804] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6289.062509] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6294.513106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6294.564534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6294.584289] Lustre: Skipped 1 previous similar message [ 6298.983900] Lustre: server umount lustre-OST0000 complete [ 6299.643057] LustreError: 189832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6299.693375] LustreError: 189832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6319.996538] Lustre: server umount lustre-OST0001 complete [ 6330.336671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6330.354421] Lustre: Skipped 4 previous similar messages [ 6334.520950] Lustre: server umount lustre-MDT0000 complete [ 6336.485655] Lustre: 178228:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779979053/real 1779979053] req@00000000459ef044 x1866442817577856/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779979069 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6336.511555] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6336.533985] Lustre: Skipped 3 previous similar messages [ 6338.533119] LustreError: 189851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779979071 with bad export cookie 16797612947552741434 [ 6338.537278] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6338.544080] LustreError: 189851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6338.853767] Lustre: server umount lustre-MDT0001 complete [ 6371.417971] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:38:22 (1779979102) [ 6391.897564] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 6403.543958] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6449.406293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6449.696524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6449.756925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6449.881548] Lustre: lustre-MDT0000: new disk, initializing [ 6450.041885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6450.063379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6455.551528] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6471.382932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6471.683586] Lustre: 195823:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6471.806784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6471.828704] Lustre: Skipped 1 previous similar message [ 6471.980379] Lustre: lustre-MDT0001: new disk, initializing [ 6472.185131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6472.199123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6478.774801] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6488.673920] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6495.709974] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6505.807575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6506.217323] Lustre: lustre-OST0000: new disk, initializing [ 6506.224207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6512.849451] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6515.826855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6515.838725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6516.069255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6521.770846] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6528.987983] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6529.183990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6534.639113] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6535.026533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6541.808966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6541.824675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6541.901747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6541.924865] Lustre: Skipped 1 previous similar message [ 6544.978660] Lustre: server umount lustre-OST0000 complete [ 6566.367647] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6566.681894] Lustre: server umount lustre-MDT0000 complete [ 6571.495097] LustreError: 195833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6571.508127] LustreError: 195833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 6572.018748] LustreError: 195814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779979305 with bad export cookie 16797612947552742995 [ 6572.036507] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6572.545358] Lustre: server umount lustre-MDT0001 complete [ 6583.290153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6590.047714] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6601.376500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6606.300629] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6614.682828] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6619.319383] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6628.410299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6634.880690] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6641.274760] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6646.469272] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6646.661753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6650.914202] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6651.173929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6655.463189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6655.467432] LustreError: Skipped 1 previous similar message [ 6659.808750] Lustre: server umount lustre-OST0000 complete [ 6667.377076] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6672.781877] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6705.055954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6706.337636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6706.357645] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6714.262500] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6721.709110] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6721.733896] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6722.235716] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6724.110258] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6732.265128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.279903] Lustre: Skipped 5 previous similar messages [ 6732.287240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6732.288907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6732.299943] LustreError: Skipped 1 previous similar message [ 6732.303664] Lustre: Skipped 11 previous similar messages [ 6734.836610] Lustre: server umount lustre-OST0000 complete [ 6745.663865] Lustre: server umount lustre-MDT0000 complete [ 6750.521720] LustreError: 199617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779979483 with bad export cookie 16797612947552743870 [ 6750.538183] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6750.539675] LustreError: 199617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6751.064162] Lustre: server umount lustre-MDT0001 complete [ 6761.299149] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:44:52 (1779979492) [ 6767.870840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6768.425750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6768.432728] Lustre: Skipped 6 previous similar messages [ 6773.039133] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6781.950435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6786.589464] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6792.179042] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6796.866752] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6805.507586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6814.254422] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6820.514716] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6825.390665] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6825.639866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6831.765638] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6832.118883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6841.095093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6841.466766] Lustre: lustre-OST0001: new disk, initializing [ 6841.486239] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6849.864076] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6850.143585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6850.156067] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6850.296968] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6861.103043] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6869.596475] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6869.994071] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6875.500476] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6875.810746] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6881.268376] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6881.278978] LustreError: Skipped 2 previous similar messages [ 6883.866337] Lustre: server umount lustre-OST0001 complete [ 6886.372407] LustreError: 209290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6886.385091] LustreError: 209290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 6890.752851] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6891.011196] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6896.319721] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6896.602560] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6932.113152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6937.587765] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6937.601844] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6937.602502] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6940.801914] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6949.162470] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6952.235988] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6958.048147] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6958.065601] LustreError: Skipped 1 previous similar message [ 6963.058892] Lustre: server umount lustre-OST0001 complete [ 6988.773837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6988.781534] Lustre: Skipped 10 previous similar messages [ 6989.302792] Lustre: server umount lustre-OST0000 complete [ 6993.888401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6993.909601] Lustre: Skipped 10 previous similar messages [ 6999.703599] Lustre: server umount lustre-MDT0000 complete [ 7003.942563] LustreError: 205074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779979736 with bad export cookie 16797612947552744836 [ 7003.964328] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7003.965516] LustreError: 205074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7010.606950] Lustre: server umount lustre-MDT0001 complete [ 7044.402299] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:49:36 (1779979776) [ 7053.018151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7053.465641] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7053.530290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7059.112540] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7070.690457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7070.767218] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7070.786937] Lustre: 213050:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7076.400421] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7085.814690] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7091.222888] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7099.081044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7099.367117] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7107.251325] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7114.494040] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7125.718327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7125.891698] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7133.260778] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7141.478210] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7144.346933] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7149.646339] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7149.904654] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7155.139406] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7155.381277] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7161.410756] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cd3085ad800.ost_server_uuid 50 [ 7163.440974] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8cd3085ad800.ost_server_uuid in IDLE state after 0 sec [ 7164.287626] Lustre: Permanently deactivating lustre-OST0001 [ 7164.305465] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7167.580402] Lustre: Permanently reactivating lustre-OST0001 [ 7173.608493] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7175.599023] Lustre: server umount lustre-OST0001 complete [ 7186.672288] Lustre: server umount lustre-OST0000 complete [ 7198.960746] Lustre: server umount lustre-MDT0000 complete [ 7204.213449] LustreError: 213043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779979937 with bad export cookie 16797612947552746131 [ 7204.222212] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7204.223307] LustreError: 213043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7204.751927] Lustre: server umount lustre-MDT0001 complete [ 7234.374910] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:52:45 (1779979965) [ 7244.434258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7244.856517] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7244.913636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7244.938355] Lustre: Skipped 5 previous similar messages [ 7251.234799] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7263.399245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7263.631819] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7263.674054] Lustre: 219169:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7268.352577] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7276.580173] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7282.362387] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7292.120732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7292.658241] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7292.730742] Lustre: Permanently deactivating lustre-OST0000 [ 7292.783248] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7293.186536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7293.190603] Lustre: Skipped 10 previous similar messages [ 7301.569159] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7302.250620] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7311.442352] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7321.648557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7321.948106] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7330.043460] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7332.092440] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7343.271992] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7347.270626] Lustre: Permanently reactivating lustre-OST0000 [ 7347.303195] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7347.312923] Lustre: Skipped 2 previous similar messages [ 7352.927035] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7352.928370] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 50 seconds. I think it's dead, and I am evicting it. exp 0000000086c10b58, cur 1779980085 deadline 1779980065 last 1779980035 [ 7352.932626] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7352.953853] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7352.956604] Lustre: Skipped 1 previous similar message [ 7352.996340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7370.798978] Lustre: server umount lustre-OST0001 complete [ 7382.397060] Lustre: server umount lustre-OST0000 complete [ 7395.650887] Lustre: server umount lustre-MDT0000 complete [ 7401.956867] LustreError: 219175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7401.991846] LustreError: 219175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 50 previous similar messages [ 7402.833575] LustreError: 219160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779980135 with bad export cookie 16797612947552747811 [ 7402.834847] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7402.852416] LustreError: 219160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7403.498377] Lustre: server umount lustre-MDT0001 complete [ 7419.195592] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 7423.841762] Key type lgssc unregistered [ 7424.174679] LNet: 223583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7424.187036] LNetError: 223583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7424.223978] LNet: Removed LNI 192.168.201.112@tcp [ 7425.251873] Key type .llcrypt unregistered [ 7425.256743] Key type ._llcrypt unregistered [ 7455.548509] Key type ._llcrypt registered [ 7455.551560] Key type .llcrypt registered [ 7455.666861] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7457.714207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7457.825168] alg: No test for adler32 (adler32-zlib) [ 7459.091868] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 7459.463581] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 7461.218488] Key type lgssc registered [ 7462.608869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7476.265945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7476.281631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.851776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7483.303857] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7487.981621] LustreError: 225036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7488.016585] LustreError: 225036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7493.094670] LustreError: 225036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7495.578656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.759620] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7495.783614] Lustre: 225023:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7495.789236] Lustre: Permanently deactivating lustre-MDT0001 [ 7495.796168] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7496.195373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7502.226349] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7505.519039] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7510.203909] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7515.136095] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7525.463695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7526.037204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7534.159619] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7541.415125] LustreError: 226656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7541.444902] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7542.749164] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7546.854731] LustreError: 227163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7548.913463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7551.664527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7551.967253] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7557.131541] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7557.144060] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7559.108825] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7566.608898] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7570.129423] Lustre: Permanently reactivating lustre-MDT0001 [ 7570.145465] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7570.149991] Lustre: Skipped 2 previous similar messages [ 7576.914606] Lustre: lustre-MDT0001: Client 921d44c9-7af8-4e3c-b832-72b43cfceb22 (at 192.168.201.12@tcp) reconnecting [ 7577.153585] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7577.162138] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7577.172811] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7577.184547] LustreError: 228160:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7577.192590] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7577.193713] LustreError: 228160:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7577.193834] LustreError: 228160:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.201.112@tcp: cfg command failed: rc = -108 [ 7577.193861] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7577.193861] [ 7577.196100] LustreError: 225028:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7582.883556] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7585.067618] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7592.061665] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7592.329865] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7597.765403] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7597.983020] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7600.845716] Lustre: Permanently deactivating lustre-MDT0001 [ 7600.859396] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7600.866712] Lustre: Skipped 2 previous similar messages [ 7609.392313] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7618.528428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7618.533182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7618.547572] Lustre: Skipped 3 previous similar messages [ 7618.560535] Lustre: Skipped 3 previous similar messages [ 7623.654899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7623.661287] Lustre: Skipped 3 previous similar messages [ 7624.611716] Lustre: server umount lustre-MDT0000 complete [ 7628.515957] LustreError: 228140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779980361 with bad export cookie 13310311840504454978 [ 7628.526357] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7628.544524] LustreError: 228140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7628.815783] Lustre: server umount lustre-MDT0001 complete [ 7635.538152] Lustre: server umount lustre-OST0000 complete [ 7650.246232] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:59:40 (1779980380) [ 7663.169965] Lustre: server umount lustre-OST0001 complete [ 7671.468160] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 7687.499871] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7735.178487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7735.419263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7735.435577] Lustre: Skipped 2 previous similar messages [ 7735.458618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7735.548660] Lustre: lustre-MDT0000: new disk, initializing [ 7735.661687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7735.686602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7740.653640] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7753.403044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7753.616762] Lustre: 232746:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7753.684629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7753.688180] Lustre: Skipped 1 previous similar message [ 7753.772832] Lustre: lustre-MDT0001: new disk, initializing [ 7753.853116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7753.894909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7753.907905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7759.964425] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7770.189404] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7775.606661] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7784.407637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7784.664240] Lustre: lustre-OST0000: new disk, initializing [ 7784.670940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7784.771783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7791.622539] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7791.680074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7791.687830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7791.859559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7801.421166] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7803.890953] LustreError: 232754:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7813.960101] LustreError: 232754:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7813.993590] LustreError: 232754:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7814.881684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7815.075631] Lustre: lustre-OST0001: new disk, initializing [ 7815.082709] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7823.050410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7823.064334] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7823.247940] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7823.533895] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 7824.064656] LustreError: 232754:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7834.227181] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7836.216666] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7843.813385] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7843.814976] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7843.839410] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7843.860934] Lustre: Skipped 1 previous similar message [ 7845.782796] Lustre: server umount lustre-OST0001 complete [ 7848.954519] LustreError: 234472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7848.969546] LustreError: 234472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7859.190263] LustreError: 234469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7859.237253] LustreError: 234469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7868.897478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7868.899820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.908906] LustreError: Skipped 2 previous similar messages [ 7868.911151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7868.927361] Lustre: Skipped 2 previous similar messages [ 7868.959267] Lustre: Skipped 1 previous similar message [ 7871.540689] Lustre: server umount lustre-OST0000 complete [ 7877.089810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7877.099251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7877.118994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7877.133674] Lustre: Skipped 2 previous similar messages [ 7882.377070] Lustre: server umount lustre-MDT0000 complete [ 7884.770669] LustreError: 232757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7884.794268] LustreError: 232757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7887.000152] LustreError: 232737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779980620 with bad export cookie 13310311840504457792 [ 7887.016313] LustreError: 232737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7887.021152] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7893.612664] Lustre: server umount lustre-MDT0001 complete [ 7907.538431] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 7912.043054] Key type lgssc unregistered [ 7912.455815] LNet: 237397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7912.467040] LNetError: 237397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7912.506810] LNet: Removed LNI 192.168.201.112@tcp [ 7914.177498] Key type .llcrypt unregistered [ 7914.179648] Key type ._llcrypt unregistered [ 7970.962470] Key type ._llcrypt registered [ 7970.965208] Key type .llcrypt registered [ 7971.074790] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 7972.923489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7972.972898] alg: No test for adler32 (adler32-zlib) [ 7974.212426] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 7974.775818] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 7976.455194] Key type lgssc registered [ 7978.017053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7988.478758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7988.529875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7989.979762] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7990.004655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7990.152809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7996.182154] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8007.899160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8008.041342] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8008.110197] Lustre: 239591:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8008.410302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8015.305122] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8023.255334] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8029.283250] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8037.943523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8038.160171] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8038.363983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8045.750430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8046.815047] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8053.179821] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8117.315238] Lustre: server umount lustre-OST0000 complete [ 8124.162283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8133.092394] Lustre: 238916:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779980850/real 1779980850] req@00000000862dd6ec x1866445049871616/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779980866 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8133.127753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8133.922399] Lustre: 238915:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779980850/real 1779980850] req@000000003077c8cb x1866445049871872/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779980866 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8133.959962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8138.208145] Lustre: 238914:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779980855/real 1779980855] req@00000000456bbf9a x1866445049873408/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779980871 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8142.820037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8143.145959] LustreError: 242846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8143.173988] LustreError: 242846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8143.311434] Lustre: 238916:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779980860/real 1779980860] req@000000001806a51b x1866445049874560/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779980876 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8143.358543] Lustre: 238916:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8143.431225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8144.452808] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8144.466516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8144.470559] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8144.486582] Lustre: Skipped 1 previous similar message [ 8151.322676] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8158.719424] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8203.678193] Lustre: server umount lustre-OST0000 complete [ 8208.352415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8208.355888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.378207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8210.921185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8210.934358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8216.034315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8216.038181] Lustre: Skipped 2 previous similar messages [ 8218.179980] Lustre: server umount lustre-MDT0000 complete [ 8221.151887] Lustre: 238917:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779980938/real 1779980938] req@00000000b659f2e1 x1866445049902976/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779980954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8221.162146] LustreError: 239603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8221.178563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8221.223390] LustreError: 239603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8222.760804] LustreError: 239584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779980955 with bad export cookie 11217979249512608307 [ 8222.773594] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8222.783507] LustreError: 239584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8223.330616] Lustre: server umount lustre-MDT0001 complete [ 8238.279523] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 8241.290902] Key type lgssc unregistered [ 8241.521758] LNet: 244555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8241.532303] LNetError: 244555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8241.546797] LNet: Removed LNI 192.168.201.112@tcp [ 8242.375067] Key type .llcrypt unregistered [ 8242.377183] Key type ._llcrypt unregistered [ 8272.473291] Key type ._llcrypt registered [ 8272.477424] Key type .llcrypt registered [ 8272.642559] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8273.967933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8274.260028] alg: No test for adler32 (adler32-zlib) [ 8275.442643] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 8275.807410] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 8277.535321] Key type lgssc registered [ 8278.945160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8287.882817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8287.912769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.508913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8294.682711] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8298.984338] LustreError: 245955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8299.018197] LustreError: 245955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8304.104946] LustreError: 245955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8304.242769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8304.788592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8309.874570] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8316.246256] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8320.830732] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8328.132419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8328.475294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8334.655584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8334.871742] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8341.705941] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8370.656767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8370.670535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.671873] LustreError: Skipped 1 previous similar message [ 8370.673595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8370.708083] Lustre: Skipped 1 previous similar message [ 8372.724733] Lustre: server umount lustre-OST0000 complete [ 8380.898382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8380.901041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8380.920756] Lustre: Skipped 1 previous similar message [ 8380.930773] Lustre: Skipped 2 previous similar messages [ 8382.804181] Lustre: server umount lustre-MDT0000 complete [ 8386.020050] LustreError: 247625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8386.043202] LustreError: 247625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8387.029042] LustreError: 245935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779981120 with bad export cookie 15234217228075248750 [ 8387.034134] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8387.039857] LustreError: 245935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8387.384241] Lustre: server umount lustre-MDT0001 complete [ 8399.642777] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 8403.155828] Key type lgssc unregistered [ 8403.572694] LNet: 249909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8403.590793] LNetError: 249909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8403.610296] LNet: Removed LNI 192.168.201.112@tcp [ 8404.685790] Key type .llcrypt unregistered [ 8404.699650] Key type ._llcrypt unregistered [ 8428.223964] Key type ._llcrypt registered [ 8428.227485] Key type .llcrypt registered [ 8428.482626] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8429.906970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8429.993065] alg: No test for adler32 (adler32-zlib) [ 8431.176926] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 8431.715743] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 8433.416516] Key type lgssc registered [ 8434.626936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8443.220715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8443.238448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8444.750828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8449.165669] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8454.113019] LustreError: 251114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8454.136725] LustreError: 251114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8459.202355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8459.233078] LustreError: 251114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8459.822562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8464.880050] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8471.650828] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8476.601477] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8484.167200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8484.534931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8490.675885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8491.454092] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8498.703799] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8511.456192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8511.471989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8511.475585] LustreError: Skipped 1 previous similar message [ 8511.491919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8511.495737] Lustre: Skipped 1 previous similar message [ 8511.525209] Lustre: Skipped 1 previous similar message [ 8516.593439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8516.603304] Lustre: Skipped 1 previous similar message [ 8517.252339] Lustre: server umount lustre-OST0000 complete [ 8521.708313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8521.732771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8526.306609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8526.315428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8526.326804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8527.550564] Lustre: server umount lustre-MDT0000 complete [ 8531.376895] LustreError: 251095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779981264 with bad export cookie 16338214066139507289 [ 8531.384770] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8531.386605] LustreError: 251095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8531.575805] Lustre: server umount lustre-MDT0001 complete [ 8546.768787] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 8551.178412] Key type lgssc unregistered [ 8551.609251] LNet: 254441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8551.628769] LNetError: 254441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8551.656794] LNet: Removed LNI 192.168.201.112@tcp [ 8552.836153] Key type .llcrypt unregistered [ 8552.839641] Key type ._llcrypt unregistered [ 8584.140515] Key type ._llcrypt registered [ 8584.141940] Key type .llcrypt registered [ 8584.292590] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8585.454726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8585.507104] alg: No test for adler32 (adler32-zlib) [ 8586.525362] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 8586.737957] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 8588.511911] Key type lgssc registered [ 8589.753838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8598.516835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8598.530504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.039063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8604.673947] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8610.275247] LustreError: 255845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8610.284284] LustreError: 255845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8613.187817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8613.611408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8619.113988] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8625.143198] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8629.920073] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8638.455215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8638.846868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8645.689296] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8646.014255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8652.406760] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8681.953619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8681.960867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8681.979467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8684.011424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8684.011750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8684.013393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8685.897142] Lustre: server umount lustre-OST0000 complete [ 8692.193633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8692.216815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8692.221340] Lustre: Skipped 1 previous similar message [ 8692.249786] Lustre: Skipped 2 previous similar messages [ 8697.174936] Lustre: server umount lustre-MDT0000 complete [ 8697.313752] LustreError: 256557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8697.348811] LustreError: 256557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8701.875519] LustreError: 259066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779981434 with bad export cookie 1924620543774630045 [ 8701.878286] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8701.884231] LustreError: 259066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8702.379538] Lustre: server umount lustre-MDT0001 complete [ 8716.712341] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 8719.571613] Key type lgssc unregistered [ 8719.932742] LNet: 259802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8719.936606] LNetError: 259802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8719.948835] LNet: Removed LNI 192.168.201.112@tcp [ 8720.768217] Key type .llcrypt unregistered [ 8720.774689] Key type ._llcrypt unregistered [ 8742.069303] Key type ._llcrypt registered [ 8742.074984] Key type .llcrypt registered [ 8742.261356] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8743.610731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8743.648546] alg: No test for adler32 (adler32-zlib) [ 8744.976360] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 8745.373665] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 8747.015544] Key type lgssc registered [ 8748.387483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8758.979955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8759.054909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8760.876535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8766.334603] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8770.022414] LustreError: 261037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8770.060912] LustreError: 261037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8775.144122] LustreError: 261037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8778.590391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.610085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8786.391790] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8795.431369] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8800.107156] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8808.389251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8808.892688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8815.782634] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 8821.582533] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8830.393448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8845.792076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8845.804960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8845.821236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8845.828362] Lustre: Skipped 1 previous similar message [ 8845.842783] Lustre: Skipped 1 previous similar message [ 8847.127452] Lustre: server umount lustre-OST0000 complete [ 8856.040483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8856.051876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8856.060421] Lustre: Skipped 1 previous similar message [ 8858.265940] Lustre: server umount lustre-MDT0000 complete [ 8861.152643] LustreError: 261015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8861.177961] LustreError: 261015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8863.167757] LustreError: 260991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779981596 with bad export cookie 7862592825803579822 [ 8863.174253] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8863.193044] LustreError: 260991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8863.799467] Lustre: server umount lustre-MDT0001 complete [ 8877.060912] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 8879.841566] Key type lgssc unregistered [ 8880.155277] LNet: 264395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8880.163581] LNetError: 264395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8881.197937] LNet: Removed LNI 192.168.201.112@tcp [ 8882.303118] Key type .llcrypt unregistered [ 8882.309037] Key type ._llcrypt unregistered [ 8900.216222] Key type ._llcrypt registered [ 8900.220360] Key type .llcrypt registered [ 8900.433096] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8919.792329] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 8937.444469] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 8939.383874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8939.420343] alg: No test for adler32 (adler32-zlib) [ 8940.401505] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 8940.558121] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 8942.295351] Key type lgssc registered [ 8943.963714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8990.226641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8990.250066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8991.536889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8991.559553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8991.635649] Lustre: lustre-MDT0000: new disk, initializing [ 8991.714786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8991.731743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8996.872122] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9007.774584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9007.882879] Lustre: 267903:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9007.921883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9007.924674] Lustre: Skipped 1 previous similar message [ 9007.975408] Lustre: lustre-MDT0001: new disk, initializing [ 9008.055641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9008.083205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9008.090262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9012.003254] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9020.472612] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9025.133628] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9033.069423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9033.325374] Lustre: lustre-OST0000: new disk, initializing [ 9033.328698] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9033.414839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9039.606635] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9042.564540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9042.594735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9042.870720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9048.088549] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9052.460846] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9052.636509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9058.137813] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9058.376881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9063.394001] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9063.396992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9063.411358] LustreError: Skipped 1 previous similar message [ 9063.415182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9063.449502] Lustre: Skipped 1 previous similar message [ 9066.911118] Lustre: server umount lustre-OST0000 complete [ 9073.633194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9073.644180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9073.650229] Lustre: Skipped 1 previous similar message [ 9073.660810] Lustre: Skipped 2 previous similar messages [ 9077.137809] Lustre: server umount lustre-MDT0000 complete [ 9081.630191] LustreError: 267892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779981814 with bad export cookie 2327223252399787934 [ 9081.646703] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9081.648308] LustreError: 267892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9082.035857] Lustre: server umount lustre-MDT0001 complete [ 9092.235685] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:23:43 (1779981823) [ 9099.744858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9100.221327] LustreError: 271910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9100.250887] LustreError: 271910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9100.322052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9105.180924] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9105.379197] LustreError: 271910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9110.503457] LustreError: 271910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9114.372396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9114.848591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9119.586435] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9127.508448] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9133.164884] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9141.076738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9141.600852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9149.817455] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9157.127913] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9169.376031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9169.387710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9169.389796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9169.416254] Lustre: Skipped 1 previous similar message [ 9174.085949] Lustre: server umount lustre-OST0000 complete [ 9179.620529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9179.635664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9179.644386] Lustre: Skipped 1 previous similar message [ 9179.662929] Lustre: Skipped 2 previous similar messages [ 9184.659765] Lustre: server umount lustre-MDT0000 complete [ 9184.740876] LustreError: 271909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9189.127118] LustreError: 271890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779981922 with bad export cookie 2327223252399788816 [ 9189.140938] LustreError: 271890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9189.154582] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9189.651447] Lustre: server umount lustre-MDT0001 complete [ 9203.686049] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 9206.557480] Key type lgssc unregistered [ 9206.888761] LNet: 275093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9206.896726] LNetError: 275093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9206.911981] LNet: Removed LNI 192.168.201.112@tcp [ 9207.745534] Key type .llcrypt unregistered [ 9207.747250] Key type ._llcrypt unregistered [ 9238.201990] Key type ._llcrypt registered [ 9238.212349] Key type .llcrypt registered [ 9238.361351] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 9256.447552] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9257.599988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9257.668990] alg: No test for adler32 (adler32-zlib) [ 9259.244610] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 9259.666584] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 9261.418825] Key type lgssc registered [ 9262.994783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9320.063358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9320.086300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9321.446908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9321.524850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9321.608304] Lustre: lustre-MDT0000: new disk, initializing [ 9321.710154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9321.741913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9327.078750] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9338.332728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9338.444460] Lustre: 278893:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9338.511508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9338.521136] Lustre: Skipped 1 previous similar message [ 9338.641092] Lustre: lustre-MDT0001: new disk, initializing [ 9338.878848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9338.947519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9338.968116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9343.865366] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9352.247739] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9355.908481] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9363.217429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9363.464682] Lustre: lustre-OST2710: new disk, initializing [ 9363.468425] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9363.548785] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9369.648920] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9369.700788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9369.708479] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9369.812229] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9377.702792] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9385.479726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9385.637319] Lustre: lustre-OST03e8: new disk, initializing [ 9385.643486] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9385.728565] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9392.242221] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9395.288611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9395.307601] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9395.446311] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9398.660683] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9406.730755] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9406.990213] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9411.780359] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9411.983765] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9417.764606] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9418.046867] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9422.731692] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9423.001297] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9429.990935] Lustre: server umount lustre-MDT0000 complete [ 9431.034858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9431.059427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9433.934092] LustreError: 278884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779982166 with bad export cookie 276926868279777128 [ 9433.958830] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9434.488432] Lustre: server umount lustre-MDT0001 complete [ 9448.654256] Lustre: server umount lustre-OST2710 complete [ 9462.703852] Lustre: server umount lustre-OST03e8 complete [ 9482.572800] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 9493.700802] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9532.043083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9532.263975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9532.287182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9532.362389] Lustre: lustre-MDT0000: new disk, initializing [ 9532.429903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9532.453588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9537.361661] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9549.960739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9550.082444] Lustre: 286618:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9550.167701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9550.173971] Lustre: Skipped 1 previous similar message [ 9550.283344] Lustre: lustre-MDT0001: new disk, initializing [ 9550.425832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9550.475642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9550.487937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9555.620989] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9564.954437] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9569.177920] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9577.680770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9578.080739] Lustre: lustre-OST0000: new disk, initializing [ 9578.086562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9578.234592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9585.422881] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9588.306782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9588.320182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9588.487514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9593.674532] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9599.735332] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9600.108513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9604.692630] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9604.971384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9609.192526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9609.199481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9609.208377] LustreError: Skipped 1 previous similar message [ 9609.215090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9609.254772] Lustre: Skipped 4 previous similar messages [ 9613.081091] Lustre: server umount lustre-OST0000 complete [ 9616.864726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9616.880225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9616.892292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9616.896294] Lustre: Skipped 1 previous similar message [ 9619.433702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9619.443191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9624.547364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9624.553897] Lustre: Skipped 2 previous similar messages [ 9629.665841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9629.671052] Lustre: Skipped 1 previous similar message [ 9631.200542] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9631.430207] Lustre: server umount lustre-MDT0000 complete [ 9634.784521] LustreError: 286625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9634.826600] LustreError: 286625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9635.448615] LustreError: 288360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779982368 with bad export cookie 276926868279778808 [ 9635.450991] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9635.463954] LustreError: 288360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9635.785849] Lustre: server umount lustre-MDT0001 complete [ 9644.531682] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:32:55 (1779982375) [ 9646.221154] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9648.199053] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:32:59 (1779982379) [ 9652.036768] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9679.668746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9679.878150] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9679.924081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9680.119967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9684.797391] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9693.710965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9693.818200] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9693.879755] Lustre: 292053:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9699.149207] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9705.045497] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9709.415396] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9727.565259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9727.898950] Lustre: Denying initial registration attempt from nid 192.168.201.112@tcp, specified as failover [ 9727.904390] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9727.914540] LustreError: 293667:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 9727.920707] LustreError: 293667:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 9727.928518] LustreError: 293667:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 9728.006765] Lustre: server umount lustre-OST0000 complete [ 9728.011088] LustreError: 293667:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9738.725748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9738.731184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9738.738096] Lustre: Skipped 1 previous similar message [ 9738.750207] Lustre: Skipped 1 previous similar message [ 9742.367768] Lustre: server umount lustre-MDT0000 complete [ 9743.840953] LustreError: 292060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9743.861838] LustreError: 292060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9746.712395] LustreError: 292045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779982479 with bad export cookie 276926868279779690 [ 9746.723815] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9747.171421] Lustre: server umount lustre-MDT0001 complete [ 9758.587865] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 9761.694612] Key type lgssc unregistered [ 9762.063632] LNet: 294772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9762.074651] LNetError: 294772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9762.103152] LNet: Removed LNI 192.168.201.112@tcp [ 9763.303072] Key type .llcrypt unregistered [ 9763.311177] Key type ._llcrypt unregistered [ 9778.591785] Key type ._llcrypt registered [ 9778.596557] Key type .llcrypt registered [ 9778.734506] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9779.994082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9780.145651] alg: No test for adler32 (adler32-zlib) [ 9781.347663] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 9781.583075] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 9783.247569] Key type lgssc registered [ 9784.759327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9847.027304] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9857.582064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9857.607932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9859.008432] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9859.023901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9859.329210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9864.935287] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9874.431345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9874.560280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9874.609134] Lustre: 297512:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9874.864825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9880.470915] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9887.391823] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9891.552399] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9910.986881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9911.241360] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9911.432559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9918.305254] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 9924.973569] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9932.261957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9932.263767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9932.270683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9932.274831] LustreError: Skipped 1 previous similar message [ 9932.302603] Lustre: Skipped 1 previous similar message [ 9933.595418] Lustre: server umount lustre-OST0000 complete [ 9941.475714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9941.488259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9941.497454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9941.512304] Lustre: Skipped 1 previous similar message [ 9942.498698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9942.507553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9945.032982] Lustre: server umount lustre-MDT0000 complete [ 9948.987029] LustreError: 297505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779982682 with bad export cookie 11007650252536527492 [ 9949.015388] LustreError: 297505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9949.023281] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9949.687968] Lustre: server umount lustre-MDT0001 complete [ 9964.010167] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 9966.962657] Key type lgssc unregistered [ 9967.289479] LNet: 300731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9967.295259] LNetError: 300731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9967.312889] LNet: Removed LNI 192.168.201.112@tcp [ 9968.082682] Key type .llcrypt unregistered [ 9968.087651] Key type ._llcrypt unregistered [ 9995.737569] Key type ._llcrypt registered [ 9995.739481] Key type .llcrypt registered [ 9995.858956] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 9996.899115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9996.991892] alg: No test for adler32 (adler32-zlib) [ 9998.112273] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [ 9998.341600] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [10000.055155] Key type lgssc registered [10001.308597] Lustre: Echo OBD driver; http://www.lustre.org/ [10010.374610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10010.438541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10012.156754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10016.677612] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10022.373220] LustreError: 302134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10022.385800] LustreError: 302134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10027.321826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10027.488667] LustreError: 302134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10027.814948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10032.281529] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10038.297179] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10042.516760] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10051.026830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10051.839991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10058.853819] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10066.050613] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10078.520389] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [10078.689921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10078.705916] Lustre: Skipped 2 previous similar messages [10081.154298] Lustre: server umount lustre-MDT0000 complete [10083.605567] LustreError: 303243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10084.615528] LustreError: 302114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779982817 with bad export cookie 6735295913412651287 [10084.620365] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10084.633832] LustreError: 302114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10088.725904] LustreError: 302131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10088.752991] LustreError: 302131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10088.759581] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [10088.767654] Lustre: Skipped 3 previous similar messages [10088.929433] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10089.958744] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10089.962378] Lustre: Skipped 1 previous similar message [10091.147809] Lustre: server umount lustre-MDT0001 complete [10097.877087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10106.843606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10109.478636] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5d7893f2b81a04fb [10109.511486] Lustre: MGC192.168.201.112@tcp: Connection restored to 0@lo (at 0@lo) [10109.839234] LustreError: 303739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10109.852669] LustreError: 303739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10109.960601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10110.029358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10114.325651] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10119.142524] LustreError: 303738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10119.155983] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10119.167193] LustreError: 303738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10123.766133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10124.243294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10129.297961] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10129.385949] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10136.038711] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10138.119948] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10142.694203] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10144.393303] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10149.859532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10149.867400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10149.877750] Lustre: Skipped 1 previous similar message [10149.878486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10149.904354] Lustre: Skipped 1 previous similar message [10154.982050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10154.991516] Lustre: Skipped 1 previous similar message [10155.763617] Lustre: server umount lustre-OST0000 complete [10160.101341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10160.120784] Lustre: Skipped 1 previous similar message [10165.222405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10165.236439] Lustre: Skipped 3 previous similar messages [10166.033443] Lustre: server umount lustre-MDT0000 complete [10169.749981] LustreError: 304578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779982902 with bad export cookie 6735295913412658427 [10169.754396] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10169.769098] LustreError: 304578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10170.379355] Lustre: server umount lustre-MDT0001 complete [10184.755573] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [10187.516312] Key type lgssc unregistered [10187.917625] LNet: 307825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10187.935305] LNetError: 307825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10187.961714] LNet: Removed LNI 192.168.201.112@tcp [10189.220843] Key type .llcrypt unregistered [10189.223686] Key type ._llcrypt unregistered [10218.380931] Key type ._llcrypt registered [10218.383791] Key type .llcrypt registered [10218.528680] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [10219.895227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10220.048057] alg: No test for adler32 (adler32-zlib) [10221.233337] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [10221.558036] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [10223.321224] Key type lgssc registered [10225.022599] Lustre: Echo OBD driver; http://www.lustre.org/ [10234.531201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10234.547398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.118596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10240.606566] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10241.510629] LustreError: 309228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10241.532642] LustreError: 309228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10246.629544] LustreError: 309228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10250.283788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10250.497228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10255.646639] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10261.183864] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10265.375429] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10271.201131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10271.201672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10271.208748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10271.222389] Lustre: Skipped 1 previous similar message [10275.142422] Lustre: server umount lustre-MDT0000 complete [10276.322777] LustreError: 309939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10276.359027] LustreError: 309939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10279.129713] LustreError: 309209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779983012 with bad export cookie 14977833862330205710 [10279.131202] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10279.138508] LustreError: 309209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10279.378544] Lustre: server umount lustre-MDT0001 complete [10293.610921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10293.954527] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10294.039754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10294.272485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10298.850609] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10308.678441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10308.802463] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10308.873871] Lustre: 311588:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10309.143816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10313.837815] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10320.482249] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10326.762309] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10337.934526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10338.439617] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10338.466573] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10338.487292] LustreError: 313247:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [10338.506574] LustreError: 313247:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [10338.509543] LustreError: 313247:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [10338.695132] Lustre: server umount lustre-OST0000 complete [10338.707351] LustreError: 313247:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -2 [10350.146206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10350.331735] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10350.676241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10355.875474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10356.965595] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10364.288654] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10374.319441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10374.474947] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10374.612545] Lustre: lustre-OST0001: new disk, initializing [10374.620469] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10374.775106] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10380.927188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10380.936236] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10381.082443] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10381.401901] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10387.870483] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10391.521383] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10391.535255] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10391.550194] LustreError: Skipped 1 previous similar message [10391.559076] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10391.559088] Lustre: Skipped 1 previous similar message [10391.589638] Lustre: Skipped 1 previous similar message [10396.647367] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10396.650895] Lustre: Skipped 2 previous similar messages [10397.541770] Lustre: server umount lustre-OST0001 complete [10401.779185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10401.783697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10401.788074] LustreError: 313713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.788086] LustreError: 313713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10401.803283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10401.855542] Lustre: Skipped 1 previous similar message [10406.888108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10406.893513] LustreError: 313714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10406.905119] Lustre: Skipped 2 previous similar messages [10406.937086] LustreError: 313714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10408.021299] Lustre: server umount lustre-OST0000 complete [10416.609643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10416.622357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10416.641664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10419.435392] Lustre: server umount lustre-MDT0000 complete [10422.241459] LustreError: 311600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10422.266563] LustreError: 311600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10423.901595] LustreError: 311580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779983156 with bad export cookie 14977833862330206270 [10423.906720] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10423.925447] LustreError: 311580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10424.489969] Lustre: server umount lustre-MDT0001 complete [10436.872827] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [10439.050286] Key type lgssc unregistered [10439.420806] LNet: 316507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10439.436938] LNetError: 316507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10439.473446] LNet: Removed LNI 192.168.201.112@tcp [10440.423025] Key type .llcrypt unregistered [10440.425274] Key type ._llcrypt unregistered [10524.921809] Key type ._llcrypt registered [10524.928165] Key type .llcrypt registered [10525.083796] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [10542.024829] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [10543.293316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10543.371495] alg: No test for adler32 (adler32-zlib) [10544.431492] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [10544.691244] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [10546.367295] Key type lgssc registered [10547.619724] Lustre: Echo OBD driver; http://www.lustre.org/ [10593.343426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10593.390361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10594.748856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10594.856949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10594.999744] Lustre: lustre-MDT0000: new disk, initializing [10595.146550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10595.176494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10600.808681] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10613.702901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10613.913469] Lustre: 321413:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10614.051893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10614.055557] Lustre: Skipped 1 previous similar message [10614.236719] Lustre: lustre-MDT0001: new disk, initializing [10614.423505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10614.507787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10614.519671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10618.771088] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10626.341430] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10630.719457] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10638.353384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10638.779387] Lustre: lustre-OST0000: new disk, initializing [10638.790374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10638.912127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10646.126308] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10648.669980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10648.692543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10648.806814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10654.426410] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10660.442425] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10660.670143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10666.340648] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10666.739240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10669.546907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10669.552245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10669.564145] LustreError: Skipped 1 previous similar message [10669.569012] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10669.575656] Lustre: Skipped 1 previous similar message [10669.595603] Lustre: Skipped 1 previous similar message [10674.666018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10674.674803] Lustre: Skipped 1 previous similar message [10679.785244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10679.805441] Lustre: Skipped 1 previous similar message [10683.359144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10683.613528] Lustre: server umount lustre-OST0000 complete [10690.016995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10690.033088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10693.775248] Lustre: server umount lustre-MDT0000 complete [10698.923572] LustreError: 321405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779983431 with bad export cookie 7913247977327308160 [10698.936181] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10698.943233] LustreError: 321405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10699.556864] Lustre: server umount lustre-MDT0001 complete [10710.482835] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:50:41 (1779983441) [10721.932487] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:50:52 (1779983452) [10730.707852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10731.102332] LustreError: 325764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10731.129192] LustreError: 325764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10731.190189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10736.611496] LustreError: 325764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10736.613203] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10741.729444] LustreError: 325764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10746.194625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10746.567438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10751.220375] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10758.194110] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10762.525874] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10769.633073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10770.031610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10776.470650] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10782.069474] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10790.840535] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10793.343546] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10795.341233] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10797.391467] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10799.362755] Lustre: Failing over lustre-MDT0000 [10799.436281] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [10799.457790] Lustre: Skipped 1 previous similar message [10799.922106] Lustre: server umount lustre-MDT0000 complete [10802.155049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10802.171715] LustreError: 325763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10802.176648] Lustre: Skipped 2 previous similar messages [10807.265862] LustreError: 325758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10807.296474] LustreError: 325758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10817.526448] LustreError: 325760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10817.554987] LustreError: 325760:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10818.527638] Lustre: 319634:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779983535/real 1779983535] req@00000000c80a9b4c x1866447744734976/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779983551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10818.573761] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10823.683554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10829.280969] LustreError: 328624:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10829.297794] LustreError: 328624:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000790472b9 x1866447744738816/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1779983561 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10829.324346] LustreError: 328624:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10830.107909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10830.145952] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10835.227128] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10835.462364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10835.522711] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10835.523886] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10835.644292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10844.601107] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10846.290562] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10850.247643] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10852.474223] Lustre: Failing over lustre-MDT0000 [10852.832358] Lustre: server umount lustre-MDT0000 complete [10855.751890] LustreError: 325760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10855.775484] LustreError: 325760:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [10855.907649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10855.912914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10855.935231] Lustre: Skipped 3 previous similar messages [10872.292957] Lustre: 319634:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779983588/real 1779983588] req@000000003e57022f x1866447744757888/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779983604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10872.319162] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10872.981758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10882.565475] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6dd1811bf42c381c [10882.599996] Lustre: MGC192.168.201.112@tcp: Connection restored to 0@lo (at 0@lo) [10882.615584] Lustre: Skipped 2 previous similar messages [10882.923825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10882.981434] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10886.447663] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10888.165809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10888.213448] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10888.269730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10888.748651] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10897.405533] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10898.946366] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10908.644317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10908.663774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10908.673036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10908.690549] Lustre: Skipped 2 previous similar messages [10912.073833] Lustre: server umount lustre-MDT0000 complete [10913.765869] LustreError: 327380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10913.793059] LustreError: 327380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [10916.475351] LustreError: 325743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779983649 with bad export cookie 7913247977327310876 [10916.483297] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10916.487162] LustreError: 325743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10917.047435] Lustre: server umount lustre-MDT0001 complete [10932.994641] Lustre: server umount lustre-OST0000 complete [10951.446583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10951.828587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10957.540481] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10968.955735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10975.068599] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [10982.815493] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10987.331383] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10997.139614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11007.184844] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11015.164049] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11021.498969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11035.107862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11035.118525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11035.129372] Lustre: Skipped 2 previous similar messages [11035.136108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11039.669195] Lustre: server umount lustre-OST0000 complete [11046.375539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11046.386192] LustreError: Skipped 1 previous similar message [11050.789840] Lustre: server umount lustre-MDT0000 complete [11052.515861] LustreError: 331559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11052.532172] LustreError: 331559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11055.245783] LustreError: 333179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779983788 with bad export cookie 7913247977327311625 [11055.248115] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11055.257000] LustreError: 333179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11055.732720] Lustre: server umount lustre-MDT0001 complete [11069.083555] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11071.971926] Key type lgssc unregistered [11072.300676] LNet: 334763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11072.318336] LNetError: 334763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11072.336776] LNet: Removed LNI 192.168.201.112@tcp [11073.195543] Key type .llcrypt unregistered [11073.197346] Key type ._llcrypt unregistered [11106.684387] Key type ._llcrypt registered [11106.688899] Key type .llcrypt registered [11106.894841] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [11109.390147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11109.430511] alg: No test for adler32 (adler32-zlib) [11110.607956] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [11110.920564] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [11112.623873] Key type lgssc registered [11114.416386] Lustre: Echo OBD driver; http://www.lustre.org/ [11124.057455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11124.098293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11125.828347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11130.944312] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11135.972172] LustreError: 336267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11135.997490] LustreError: 336267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11140.520762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11140.910472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11145.790252] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11152.862420] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11158.198466] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11166.445282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11166.992984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11170.139420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11174.309407] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11181.780423] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11197.411950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11197.423283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11197.450637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11200.993938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11201.009962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11203.032368] Lustre: server umount lustre-MDT0000 complete [11206.115121] LustreError: 337882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11206.160889] LustreError: 337882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11207.355627] LustreError: 336248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779983940 with bad export cookie 2056824966048826532 [11207.359867] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11207.365376] LustreError: 336248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11207.783410] Lustre: server umount lustre-MDT0001 complete [11224.667375] Lustre: server umount lustre-OST0000 complete [11242.853868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11243.420681] LustreError: 339670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11243.577809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11248.613824] LustreError: 339670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11248.803422] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11253.728409] LustreError: 339670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11259.860571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11260.435154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11266.203664] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11273.821945] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11280.811587] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11291.298691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11291.909394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11299.610676] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11306.512318] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11313.393343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11332.065646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11332.085203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11332.103312] Lustre: Skipped 1 previous similar message [11332.112280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11332.115764] Lustre: Skipped 2 previous similar messages [11334.124748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11334.128856] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11334.135104] Lustre: Skipped 1 previous similar message [11334.150688] Lustre: Skipped 1 previous similar message [11337.318727] Lustre: server umount lustre-MDT0000 complete [11339.252852] LustreError: 339666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11339.293826] LustreError: 339666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11341.703875] LustreError: 339652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779984074 with bad export cookie 2056824966048827645 [11341.707551] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11341.715408] LustreError: 339652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11342.109118] Lustre: server umount lustre-MDT0001 complete [11356.611153] Lustre: server umount lustre-OST0000 complete [11374.756524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11375.382081] LustreError: 343284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11375.450925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11380.296357] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11391.068591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11397.930577] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11406.318559] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11411.659944] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11420.672769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11421.192559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11421.198233] Lustre: Skipped 1 previous similar message [11427.811925] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11435.198210] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11445.756939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11452.897208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11452.905853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11452.917960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11452.925522] Lustre: Skipped 1 previous similar message [11461.089725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11461.101416] Lustre: Skipped 3 previous similar messages [11467.231238] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11467.485743] Lustre: server umount lustre-OST0000 complete [11473.377990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11473.382914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11473.397781] Lustre: Skipped 1 previous similar message [11478.319965] Lustre: server umount lustre-MDT0000 complete [11481.573346] LustreError: 343283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.602125] LustreError: 343283:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11483.213268] LustreError: 343264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779984216 with bad export cookie 2056824966048828667 [11483.237320] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11483.584823] Lustre: server umount lustre-MDT0001 complete [11496.528361] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11499.519731] Key type lgssc unregistered [11499.811038] LNet: 346486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11499.819379] LNetError: 346486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11499.838524] LNet: Removed LNI 192.168.201.112@tcp [11500.871727] Key type .llcrypt unregistered [11500.873896] Key type ._llcrypt unregistered [11534.750361] Key type ._llcrypt registered [11534.752500] Key type .llcrypt registered [11534.862519] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [11536.137653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11536.403518] alg: No test for adler32 (adler32-zlib) [11537.458267] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [11537.663822] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [11539.447822] Key type lgssc registered [11540.666190] Lustre: Echo OBD driver; http://www.lustre.org/ [11548.378895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11548.405699] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11548.415331] LustreError: 347913:0:(osd_handler.c:8556:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11548.437521] LustreError: 347913:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [11548.450239] LustreError: 347913:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11548.458835] LustreError: 347913:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11548.466558] LustreError: 347913:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [11556.855120] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11556.864611] LustreError: 348311:0:(osd_handler.c:8556:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11556.879283] LustreError: 348311:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [11556.897814] LustreError: 348311:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11556.910648] LustreError: 348311:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11556.922610] LustreError: 348311:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [11580.358457] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11583.706486] Key type lgssc unregistered [11584.086679] LNet: 349299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11584.093536] LNetError: 349299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11584.127243] LNet: Removed LNI 192.168.201.112@tcp [11585.015347] Key type .llcrypt unregistered [11585.017896] Key type ._llcrypt unregistered [11604.824304] Key type ._llcrypt registered [11604.826367] Key type .llcrypt registered [11604.925544] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [11622.581896] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [11623.948908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11624.156742] alg: No test for adler32 (adler32-zlib) [11625.248855] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [11625.460895] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [11627.087164] Key type lgssc registered [11628.291162] Lustre: Echo OBD driver; http://www.lustre.org/ [11670.434420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11670.527141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11671.938283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11672.029924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11672.189586] Lustre: lustre-MDT0000: new disk, initializing [11672.360052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11672.379205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11676.835563] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11688.251131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11688.412557] Lustre: 352426:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11688.487699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11688.501502] Lustre: Skipped 1 previous similar message [11688.609475] Lustre: lustre-MDT0001: new disk, initializing [11688.726367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11688.794059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11688.814647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11694.166990] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11703.277703] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11707.605351] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11716.213451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11716.505435] Lustre: lustre-OST0000: new disk, initializing [11716.510428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11716.594117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11723.412458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11723.442302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11723.587909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11724.027920] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11733.835077] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11739.673572] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11739.972573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11745.365673] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11745.564321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11749.344196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11749.349632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11749.380313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11749.389863] Lustre: Skipped 1 previous similar message [11754.473979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11754.488401] Lustre: Skipped 1 previous similar message [11759.588569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11759.595875] Lustre: Skipped 1 previous similar message [11762.143224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11762.679450] Lustre: server umount lustre-OST0000 complete [11769.825821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11769.868666] Lustre: Skipped 1 previous similar message [11769.889965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11769.898885] Lustre: Skipped 1 previous similar message [11773.791426] Lustre: server umount lustre-MDT0000 complete [11778.586098] LustreError: 352419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779984511 with bad export cookie 9659221644945864793 [11778.600394] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11778.607996] LustreError: 352419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11779.172079] Lustre: server umount lustre-MDT0001 complete [11790.389859] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11792.338876] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:08:43 (1779984523) [11800.250272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11801.105834] LustreError: 356483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.133318] LustreError: 356483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11801.248535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11806.690669] LustreError: 356483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11807.326624] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11811.813962] LustreError: 356483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11816.946299] LustreError: 356483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11821.120402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11821.921497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11827.909794] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11835.188586] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11840.016638] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11848.155849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11848.456562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11855.717123] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11863.460380] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11871.422582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11871.645805] Lustre: lustre-OST0001: new disk, initializing [11871.649451] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11871.742622] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11879.205318] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [11879.675294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11879.696529] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11879.766514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11889.195188] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11895.264775] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11895.276343] LustreError: Skipped 1 previous similar message [11895.285897] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11895.302824] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11895.307047] Lustre: Skipped 2 previous similar messages [11895.324560] Lustre: Skipped 1 previous similar message [11899.908508] Lustre: server umount lustre-OST0001 complete [11900.387710] LustreError: 358081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11900.397643] LustreError: 358081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11910.629695] LustreError: 358078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11910.662929] LustreError: 358078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11918.817546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11918.827716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11918.842665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11918.847987] Lustre: Skipped 1 previous similar message [11924.925214] Lustre: server umount lustre-OST0000 complete [11929.568304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11929.579591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11929.600675] Lustre: Skipped 1 previous similar message [11935.686560] Lustre: server umount lustre-MDT0000 complete [11936.227325] LustreError: 356482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11936.252823] LustreError: 356482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11939.519815] LustreError: 356464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779984672 with bad export cookie 9659221644945865675 [11939.520994] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11939.530173] LustreError: 356464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11946.041193] Lustre: server umount lustre-MDT0001 complete [11959.230287] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [11962.064524] Key type lgssc unregistered [11962.442368] LNet: 360845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11962.458858] LNetError: 360845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11963.504042] LNet: Removed LNI 192.168.201.112@tcp [11964.398607] Key type .llcrypt unregistered [11964.403480] Key type ._llcrypt unregistered [12024.595930] Key type ._llcrypt registered [12024.598134] Key type .llcrypt registered [12024.706726] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12025.833982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12025.902074] alg: No test for adler32 (adler32-zlib) [12027.082960] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [12027.394560] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12029.063199] Key type lgssc registered [12030.624583] Lustre: Echo OBD driver; http://www.lustre.org/ [12039.391530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12039.428808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12040.785441] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12040.827831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12041.032672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12045.927752] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12056.015986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12056.172539] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12056.216240] Lustre: 363629:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12056.596829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12061.874812] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12069.637626] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12074.955400] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12079.587152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12079.604686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12079.614840] Lustre: Skipped 1 previous similar message [12083.992524] Lustre: server umount lustre-MDT0000 complete [12088.549333] LustreError: 363622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779984821 with bad export cookie 5602615280969775425 [12088.556183] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12088.558069] LustreError: 363622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12089.101956] Lustre: server umount lustre-MDT0001 complete [12100.854371] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:13:52 (1779984832) [12111.250207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12111.829285] LustreError: 365916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12111.843927] LustreError: 365916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12111.887698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12116.979320] LustreError: 365916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12119.325534] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12122.081377] LustreError: 365916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12127.206838] LustreError: 365916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12132.105821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12132.322165] LustreError: 365916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12132.613272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12138.630459] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12145.648561] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12150.475598] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12158.870697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12159.135101] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12159.321868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12167.916506] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12176.574649] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12189.242836] Lustre: Permanently deactivating lustre-OST0000 [12189.256361] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12194.915941] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12196.374941] LustreError: 368467:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12196.386639] LustreError: 368467:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12196.400912] LustreError: 368467:0:(mgs_handler.c:1208:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12199.688755] LustreError: 368515:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12199.693591] LustreError: 368515:0:(mgs_handler.c:1208:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12208.284436] Lustre: server umount lustre-OST0000 complete [12211.525974] LustreError: 368765:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12211.535874] LustreError: 368765:0:(mgs_handler.c:1208:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12214.755613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12214.758527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12214.784641] Lustre: Skipped 1 previous similar message [12214.786798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12219.873923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12219.878528] Lustre: Skipped 2 previous similar messages [12224.993489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12225.005738] Lustre: Skipped 1 previous similar message [12228.063118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12228.316423] Lustre: server umount lustre-MDT0000 complete [12230.119080] LustreError: 367119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12230.141288] LustreError: 367119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12232.171861] LustreError: 367522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779984965 with bad export cookie 5602615280969776006 [12232.178411] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12232.184611] LustreError: 367522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12242.648812] Lustre: server umount lustre-MDT0001 complete [12254.195641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12259.962700] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12263.827763] LustreError: 369912:0:(mgs_handler.c:1208:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12271.693505] Lustre: 370251:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.112@tcp [12274.476633] Lustre: server umount MGS complete [12289.170824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12289.818500] LustreError: 370977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12289.854587] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12289.862949] Lustre: Skipped 1 previous similar message [12289.973897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12296.328486] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12307.349883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12307.612280] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12307.696928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12313.581738] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12319.661620] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12325.865492] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12328.688717] Lustre: Permanently reactivating lustre-OST0000 [12328.753342] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12328.756118] Lustre: Skipped 2 previous similar messages [12355.047277] LustreError: 372453:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12366.061039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12366.681027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12374.143973] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12380.790849] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12390.368748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12390.374468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12390.382439] Lustre: Skipped 1 previous similar message [12390.393404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12391.392293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12393.503782] Lustre: server umount lustre-OST0000 complete [12400.099062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12400.109182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12400.130560] Lustre: Skipped 1 previous similar message [12400.142264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12400.150307] Lustre: Skipped 2 previous similar messages [12403.916466] Lustre: server umount lustre-MDT0000 complete [12406.753523] LustreError: 370972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12406.789700] LustreError: 370972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12407.652935] LustreError: 370956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779985140 with bad export cookie 5602615280969777049 [12407.660865] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12407.673877] LustreError: 370956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12408.134535] Lustre: server umount lustre-MDT0001 complete [12420.936450] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12424.816975] Key type lgssc unregistered [12425.180845] LNet: 374892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12425.189548] LNetError: 374892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12425.209549] LNet: Removed LNI 192.168.201.112@tcp [12427.105750] Key type .llcrypt unregistered [12427.111835] Key type ._llcrypt unregistered [12453.245570] Key type ._llcrypt registered [12453.248153] Key type .llcrypt registered [12453.357607] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [12473.919267] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12475.657351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12475.771829] alg: No test for adler32 (adler32-zlib) [12477.684867] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [12478.201700] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12479.975203] Key type lgssc registered [12482.943464] Lustre: Echo OBD driver; http://www.lustre.org/ [12527.693150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12527.734706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12528.989248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12529.018574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12529.068854] Lustre: lustre-MDT0000: new disk, initializing [12529.119712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12529.136794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12533.618164] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12545.759599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12545.911452] Lustre: 378019:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12546.059792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12546.067398] Lustre: Skipped 1 previous similar message [12546.223625] Lustre: lustre-MDT0001: new disk, initializing [12546.419172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12546.476545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12546.495543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12551.862300] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12560.842691] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12566.324754] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12576.052557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12576.292657] Lustre: lustre-OST0000: new disk, initializing [12576.295492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12576.403727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12583.008715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12583.019855] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12583.287243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12587.418697] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12599.569131] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12604.640372] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12604.911861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12610.069271] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12610.396738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12614.112917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12614.117265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12614.121472] LustreError: Skipped 1 previous similar message [12614.126440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12614.136438] Lustre: Skipped 1 previous similar message [12614.165548] Lustre: Skipped 1 previous similar message [12619.235707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12619.660569] Lustre: server umount lustre-OST0000 complete [12624.353917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12624.370281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12624.381322] Lustre: Skipped 1 previous similar message [12624.406642] Lustre: Skipped 2 previous similar messages [12629.473532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12630.087569] Lustre: server umount lustre-MDT0000 complete [12634.177884] LustreError: 379762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779985367 with bad export cookie 1451806292664355618 [12634.187929] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12634.190506] LustreError: 379762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12634.849160] Lustre: server umount lustre-MDT0001 complete [12645.634219] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12648.296962] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:22:59 (1779985379) [12657.919659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12658.858755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12664.290280] LustreError: 382079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12664.323388] LustreError: 382079:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12664.796600] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12668.397404] LustreError: 382079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12673.526852] LustreError: 382079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12677.277634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12677.538592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12683.406116] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12691.394126] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12696.639445] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12706.603029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12707.484960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12717.709833] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12727.696158] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12730.343540] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12734.907890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12743.650575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12743.656192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12743.670132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12743.680381] Lustre: Skipped 2 previous similar messages [12747.292603] Lustre: server umount lustre-OST0000 complete [12753.890989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12753.895055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12753.903513] Lustre: Skipped 2 previous similar messages [12753.921524] Lustre: Skipped 1 previous similar message [12757.636519] Lustre: server umount lustre-MDT0000 complete [12759.009522] LustreError: 382790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12759.041860] LustreError: 382790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12762.303872] LustreError: 382060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779985495 with bad export cookie 1451806292664356507 [12762.307228] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12762.319020] LustreError: 382060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12762.707047] Lustre: server umount lustre-MDT0001 complete [12778.029913] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12782.351264] Key type lgssc unregistered [12782.788781] LNet: 385454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12782.792880] LNetError: 385454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12782.803858] LNet: Removed LNI 192.168.201.112@tcp [12783.637976] Key type .llcrypt unregistered [12783.640513] Key type ._llcrypt unregistered [12813.171834] Key type ._llcrypt registered [12813.173939] Key type .llcrypt registered [12813.281292] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12813.633515] Key type .llcrypt unregistered [12813.636236] Key type ._llcrypt unregistered [12833.744949] Key type ._llcrypt registered [12833.749972] Key type .llcrypt registered [12833.831787] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12835.098805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12835.166767] alg: No test for adler32 (adler32-zlib) [12836.339958] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [12836.542665] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12838.223227] Key type lgssc registered [12839.780930] Lustre: Echo OBD driver; http://www.lustre.org/ [12848.672755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12848.691167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12850.422454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12855.627444] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12860.903072] LustreError: 387635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12860.932498] LustreError: 387635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12866.023088] LustreError: 387635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12866.443765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12866.837851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12872.161898] LustreError: 387629:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12872.198228] LustreError: 387629:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12875.176278] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12875.252366] LustreError: 387634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12875.277161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12875.285995] LustreError: 387634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12880.355185] LustreError: 387634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12880.393488] LustreError: 387634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12885.845369] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12890.594780] LustreError: 387631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12890.672667] LustreError: 387631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12896.877636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12897.690109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12901.630836] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [12911.077321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12911.081693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12911.105975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12913.551723] Lustre: server umount lustre-OST0000 complete [12921.318899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12921.327672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12921.338929] Lustre: Skipped 2 previous similar messages [12921.363855] Lustre: Skipped 2 previous similar messages [12925.584133] Lustre: server umount lustre-MDT0000 complete [12926.433288] LustreError: 387634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12926.455881] LustreError: 387634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12930.319196] LustreError: 389689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779985663 with bad export cookie 3418758935402502162 [12930.324463] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12930.334429] LustreError: 389689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12930.794372] Lustre: server umount lustre-MDT0001 complete [12945.892205] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [12949.641889] Key type lgssc unregistered [12950.052544] LNet: 390627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12950.058513] LNetError: 390627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12950.072313] LNet: Removed LNI 192.168.201.112@tcp [12950.812306] Key type .llcrypt unregistered [12950.813684] Key type ._llcrypt unregistered [12981.173256] Key type ._llcrypt registered [12981.175622] Key type .llcrypt registered [12981.337905] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [12982.604235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12982.830545] alg: No test for adler32 (adler32-zlib) [12983.984852] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [12984.206773] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [12985.856070] Key type lgssc registered [12986.974385] Lustre: Echo OBD driver; http://www.lustre.org/ [12995.108223] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12995.143870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13010.719445] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13015.839521] LustreError: 392001:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [13042.591259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13050.588871] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13058.468460] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13067.268684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13067.641776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13067.711753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13071.127610] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13072.878078] LustreError: 392026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13072.900745] LustreError: 392026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13077.994867] LustreError: 392026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13078.027720] LustreError: 392026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13079.986182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13080.553355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13087.104854] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13093.019986] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13098.163961] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13105.843546] Lustre: server umount lustre-OST0000 complete [13111.265735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13111.277311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13111.291954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13111.316056] Lustre: Skipped 1 previous similar message [13116.387119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13116.391849] Lustre: Skipped 2 previous similar messages [13119.677802] Lustre: server umount lustre-MDT0000 complete [13121.509054] LustreError: 392930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13121.533142] LustreError: 392930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13122.527469] Lustre: 391341:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779985839/real 1779985839] req@00000000e079a809 x1866450303217664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779985855 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13122.576100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13124.250937] LustreError: 392007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779985857 with bad export cookie 5201842661677233392 [13124.281798] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13125.020392] Lustre: server umount lustre-MDT0001 complete [13138.908469] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13142.766887] Key type lgssc unregistered [13143.135049] LNet: 395217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13143.146679] LNetError: 395217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13143.191688] LNet: Removed LNI 192.168.201.112@tcp [13144.349776] Key type .llcrypt unregistered [13144.363309] Key type ._llcrypt unregistered [13184.440830] Key type ._llcrypt registered [13184.444039] Key type .llcrypt registered [13184.583066] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [13185.891154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13185.940224] alg: No test for adler32 (adler32-zlib) [13186.981758] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [13187.140363] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [13188.815356] Key type lgssc registered [13190.090805] Lustre: Echo OBD driver; http://www.lustre.org/ [13199.282608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13199.308634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13201.377627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13207.176273] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13211.653744] LustreError: 396619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13211.702390] LustreError: 396619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13216.736767] LustreError: 396619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13219.682049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13220.431375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13225.675425] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13232.371861] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13237.351596] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13245.526315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13245.923443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13246.968108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13253.274412] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13261.526214] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13267.425204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13267.432154] Lustre: Skipped 2 previous similar messages [13267.438518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13269.385673] Lustre: server umount lustre-MDT0000 complete [13269.831273] LustreError: 396616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13271.902647] LustreError: 396600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.12@tcp arrived at 1779986004 with bad export cookie 3029448629396697418 [13271.934929] LustreError: 396600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13272.547660] LustreError: 396619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13272.582286] LustreError: 396619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13274.692413] LustreError: 396600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779986007 with bad export cookie 3029448629396696522 [13274.717919] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13274.721641] LustreError: 396600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13277.665708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13277.667381] LustreError: 396614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13277.710852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13277.710861] Lustre: Skipped 2 previous similar messages [13277.736406] LustreError: 396614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13279.722766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13281.067950] Lustre: server umount lustre-OST0000 complete [13288.487879] Lustre: server umount lustre-MDT0001 complete [13303.730881] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13307.081058] Key type lgssc unregistered [13307.312650] LNet: 399954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13307.324896] LNetError: 399954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13308.390438] LNet: Removed LNI 192.168.201.112@tcp [13309.745104] Key type .llcrypt unregistered [13309.746777] Key type ._llcrypt unregistered [13343.316392] Key type ._llcrypt registered [13343.318548] Key type .llcrypt registered [13343.489303] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [13344.657134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13344.691163] alg: No test for adler32 (adler32-zlib) [13346.017793] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [13346.330084] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [13348.063568] Key type lgssc registered [13349.503854] Lustre: Echo OBD driver; http://www.lustre.org/ [13359.991824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13360.020442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13361.928701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13367.253420] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13371.363782] LustreError: 401356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13371.382318] LustreError: 401356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13376.487761] LustreError: 401356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13377.409669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13378.229652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13382.893683] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13389.663903] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13394.867456] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13402.546821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13402.817646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13406.924697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13409.533448] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13416.343888] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13422.561504] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13422.592830] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13424.468125] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [13424.483203] Lustre: Skipped 1 previous similar message [13427.680030] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13427.905786] Lustre: server umount lustre-MDT0001 complete [13429.537863] LustreError: 403513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13432.810073] LustreError: 403513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13432.837499] LustreError: 403513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13437.921425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13437.937960] LustreError: 401356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13437.954733] Lustre: Skipped 1 previous similar message [13437.963981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13437.989446] Lustre: Skipped 1 previous similar message [13441.334118] Lustre: server umount lustre-OST0000 complete [13446.336192] Lustre: server umount lustre-MDT0000 complete [13464.540032] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13467.379414] Key type lgssc unregistered [13467.752597] LNet: 404691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13467.762516] LNetError: 404691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13467.786406] LNet: Removed LNI 192.168.201.112@tcp [13468.887773] Key type .llcrypt unregistered [13468.889940] Key type ._llcrypt unregistered [13497.956184] Key type ._llcrypt registered [13497.957855] Key type .llcrypt registered [13498.158872] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [13517.466824] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [13518.573622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13518.627813] alg: No test for adler32 (adler32-zlib) [13519.970840] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [13520.238701] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [13522.031229] Key type lgssc registered [13523.448804] Lustre: Echo OBD driver; http://www.lustre.org/ [13567.944091] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [13587.283175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13587.358925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13588.852282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13588.906827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13589.046811] Lustre: lustre-MDT0000: new disk, initializing [13589.171690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13589.205082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13594.107604] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13618.267310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13618.829335] Lustre: lustre-OST0000: new disk, initializing [13618.836230] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13618.847496] Lustre: Skipped 1 previous similar message [13619.074525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13624.515160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13624.530239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13624.604686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13626.650611] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13636.517618] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13655.762527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13655.991267] Lustre: 408657:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13656.092302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13656.186605] Lustre: lustre-MDT0001: new disk, initializing [13656.363199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13656.453479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13656.465484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13661.640459] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13672.792732] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13673.047858] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13678.123960] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13678.288884] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13684.193298] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13684.204445] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13684.211682] Lustre: Skipped 1 previous similar message [13685.566857] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [13689.322298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13689.327722] Lustre: Skipped 1 previous similar message [13689.988579] Lustre: server umount lustre-MDT0001 complete [13699.557932] LustreError: 408668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13699.591287] LustreError: 408668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [13704.676873] LustreError: 408668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13704.709201] LustreError: 408668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13709.807220] LustreError: 408664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13709.834556] LustreError: 408664:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13711.840874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13711.863925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13711.886421] Lustre: Skipped 1 previous similar message [13711.904424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13714.912524] LustreError: 408669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13720.033591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13720.033624] LustreError: 408669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13720.038950] Lustre: Skipped 1 previous similar message [13724.132865] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13724.734534] Lustre: server umount lustre-OST0000 complete [13735.724139] Lustre: server umount lustre-MDT0000 complete [13752.798094] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13756.099654] Key type lgssc unregistered [13756.436666] LNet: 413233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13756.439687] LNetError: 413233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13757.483901] LNet: Removed LNI 192.168.201.112@tcp [13758.741622] Key type .llcrypt unregistered [13758.742941] Key type ._llcrypt unregistered [13810.501391] Key type ._llcrypt registered [13810.508977] Key type .llcrypt registered [13810.624498] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [13811.101509] Key type .llcrypt unregistered [13811.107419] Key type ._llcrypt unregistered [13830.425118] Key type ._llcrypt registered [13830.428360] Key type .llcrypt registered [13830.544485] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [13831.449652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13831.493222] alg: No test for adler32 (adler32-zlib) [13832.933373] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [13833.309346] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [13835.119174] Key type lgssc registered [13836.399905] Lustre: Echo OBD driver; http://www.lustre.org/ [13872.733914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13872.784427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13874.230553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13874.346674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13874.563691] Lustre: lustre-MDT0000: new disk, initializing [13874.686980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13874.708211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13879.396173] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13892.145938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13892.320980] Lustre: 416587:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13892.378601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13892.382610] Lustre: Skipped 1 previous similar message [13892.495506] Lustre: lustre-MDT0001: new disk, initializing [13892.694578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13892.755276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13892.767058] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13899.042133] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13908.817885] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13913.671889] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13921.525598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13921.934792] Lustre: lustre-OST0000: new disk, initializing [13921.944157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13922.079139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13929.191115] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13932.164352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13932.204187] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13932.364960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13938.116420] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13947.889424] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13947.901250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13947.908064] Lustre: Skipped 2 previous similar messages [13947.929828] Lustre: Skipped 2 previous similar messages [13951.941116] Lustre: server umount lustre-MDT0000 complete [13956.645760] LustreError: 416579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779986689 with bad export cookie 9132712446259215075 [13956.662176] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13956.934160] Lustre: server umount lustre-MDT0001 complete [13971.566365] Lustre: server umount lustre-OST0000 complete [13984.921425] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:45:16 (1779986716) [13993.476670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13994.167317] LustreError: 420244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13994.204974] LustreError: 420244:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13994.303151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13998.971245] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [13999.586076] LustreError: 420244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14004.706902] LustreError: 420244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14009.456400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14009.798556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14014.432955] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14021.596351] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14025.861560] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14044.978770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14045.392469] Lustre: Found index 0 for lustre-OST0000, updating log [14045.623776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14052.789415] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14059.056914] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14064.062812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14070.245772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14070.258536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14070.291980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14071.338988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14071.348355] Lustre: Skipped 2 previous similar messages [14075.608687] Lustre: server umount lustre-OST0000 complete [14092.773748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14092.777471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14092.797686] Lustre: Skipped 2 previous similar messages [14092.807776] Lustre: Skipped 1 previous similar message [14097.897796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14097.908675] Lustre: Skipped 1 previous similar message [14098.380209] Lustre: server umount lustre-MDT0000 complete [14098.922460] LustreError: 420240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14098.934830] LustreError: 420240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14102.519817] LustreError: 420225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779986835 with bad export cookie 9132712446259216475 [14102.528179] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14102.530537] LustreError: 420225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [14102.988438] Lustre: server umount lustre-MDT0001 complete [14110.815482] LNet: 1 local NIs in recovery (showing 1): 192.168.201.112@tcp [14112.521590] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:47:23 (1779986843) [14741.442922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14742.116033] LustreError: 426323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14742.136281] LustreError: 426323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14742.235718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14747.113831] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14747.629468] LustreError: 426323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14752.754913] LustreError: 426323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14757.297991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14757.732157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14762.538496] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14768.755667] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14773.003494] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14790.622699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14790.832883] Lustre: Found index 0 for lustre-OST0000, updating log [14791.117675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14797.599816] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [14800.996112] LNet: 1 local NIs in recovery (showing 1): 192.168.201.112@tcp [14803.617907] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14809.573705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14809.575162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14809.599631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14812.025393] Lustre: server umount lustre-OST0000 complete [14816.750948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14816.758040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14816.764737] Lustre: Skipped 1 previous similar message [14816.785697] Lustre: Skipped 1 previous similar message [14818.849434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14818.859419] Lustre: Skipped 3 previous similar messages [14822.094095] Lustre: server umount lustre-MDT0000 complete [14822.956650] LustreError: 426323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14822.985294] LustreError: 426323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14825.893503] LustreError: 428785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779987558 with bad export cookie 9132712446259217560 [14825.898568] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14825.902070] LustreError: 428785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14826.255561] Lustre: server umount lustre-MDT0001 complete [14858.873278] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [14865.503301] LNet: 1 local NIs in recovery (showing 1): 192.168.201.112@tcp [14869.017863] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [14916.716739] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:00:47 (1779987647) [14935.393704] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [14937.452783] Key type lgssc unregistered [14937.735726] LNet: 432560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14937.761323] LNetError: 432560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15054.507306] LNet: Removed LNI 192.168.201.112@tcp [15055.674158] Key type .llcrypt unregistered [15055.677625] Key type ._llcrypt unregistered [15071.740939] Key type ._llcrypt registered [15071.744577] Key type .llcrypt registered [15071.851623] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [15073.360768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15073.396597] alg: No test for adler32 (adler32-zlib) [15074.580821] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [15074.829061] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [15076.487198] Key type lgssc registered [15078.227535] Lustre: Echo OBD driver; http://www.lustre.org/ [15082.360639] LNet: Added LNI 192.168.201.112@tcp100 [8/256/0/180] [15083.196491] LNet: Added LNI 192.168.201.112@tcp101 [8/256/0/180] [15084.069994] LNet: Added LNI 192.168.201.112@tcp102 [8/256/0/180] [15084.945883] LNet: Added LNI 192.168.201.112@tcp103 [8/256/0/180] [15085.757030] LNet: Added LNI 192.168.201.112@tcp104 [8/256/0/180] [15086.561453] LNet: Added LNI 192.168.201.112@tcp105 [8/256/0/180] [15087.348820] LNet: Added LNI 192.168.201.112@tcp106 [8/256/0/180] [15088.104338] LNet: Added LNI 192.168.201.112@tcp107 [8/256/0/180] [15088.807350] LNet: Added LNI 192.168.201.112@tcp108 [8/256/0/180] [15089.707266] LNet: Added LNI 192.168.201.112@tcp109 [8/256/0/180] [15090.568693] LNet: Added LNI 192.168.201.112@tcp110 [8/256/0/180] [15091.300860] LNet: Added LNI 192.168.201.112@tcp111 [8/256/0/180] [15092.117197] LNet: Added LNI 192.168.201.112@tcp112 [8/256/0/180] [15093.100913] LNet: Added LNI 192.168.201.112@tcp113 [8/256/0/180] [15094.140289] LNet: Added LNI 192.168.201.112@tcp114 [8/256/0/180] [15095.096448] LNet: Added LNI 192.168.201.112@tcp115 [8/256/0/180] [15095.932784] LNet: Added LNI 192.168.201.112@tcp116 [8/256/0/180] [15096.715784] LNet: Added LNI 192.168.201.112@tcp117 [8/256/0/180] [15097.547465] LNet: Added LNI 192.168.201.112@tcp118 [8/256/0/180] [15098.505485] LNet: Added LNI 192.168.201.112@tcp119 [8/256/0/180] [15099.400985] LNet: Added LNI 192.168.201.112@tcp120 [8/256/0/180] [15100.272640] LNet: Added LNI 192.168.201.112@tcp121 [8/256/0/180] [15101.073509] LNet: Added LNI 192.168.201.112@tcp122 [8/256/0/180] [15101.900149] LNet: Added LNI 192.168.201.112@tcp123 [8/256/0/180] [15102.871660] LNet: Added LNI 192.168.201.112@tcp124 [8/256/0/180] [15103.738950] LNet: Added LNI 192.168.201.112@tcp125 [8/256/0/180] [15104.792136] LNet: Added LNI 192.168.201.112@tcp126 [8/256/0/180] [15105.800708] LNet: Added LNI 192.168.201.112@tcp127 [8/256/0/180] [15106.651930] LNet: Added LNI 192.168.201.112@tcp128 [8/256/0/180] [15107.939832] LNet: Added LNI 192.168.201.112@tcp129 [8/256/0/180] [15109.000560] LNet: Added LNI 192.168.201.112@tcp130 [8/256/0/180] [15109.861113] LNet: Added LNI 192.168.201.112@tcp131 [8/256/0/180] [15110.826591] LNet: Added LNI 192.168.201.112@tcp132 [8/256/0/180] [15112.002909] LNet: Added LNI 192.168.201.112@tcp133 [8/256/0/180] [15113.336769] LNet: Added LNI 192.168.201.112@tcp134 [8/256/0/180] [15114.645509] LNet: Added LNI 192.168.201.112@tcp135 [8/256/0/180] [15126.004363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15126.040178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15127.259360] Lustre: 435578:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.112@tcp: can't add NID 192.168.201.112@tcp130: rc = -75 [15127.536715] LNetError: 435608:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [15127.581992] LNetError: Protocol error connecting to 192.168.201.112@tcp110 at host 192.168.201.112:988: is it running a compatible version of Lustre? [15127.845781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15127.910038] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15128.052511] Lustre: lustre-MDT0000: new disk, initializing [15128.105366] LNet: 5 local NIs in recovery (showing 5): 192.168.201.112@tcp125, 192.168.201.112@tcp124, 192.168.201.112@tcp127, 192.168.201.112@tcp129, 192.168.201.112@tcp128 [15128.109705] LNetError: 433037:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [15128.152994] LNetError: 433037:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 10 previous similar messages [15128.185398] LNetError: Protocol error connecting to 192.168.201.112@tcp101 at host 192.168.201.112:988: is it running a compatible version of Lustre? [15128.217727] LNetError: Skipped 10 previous similar messages [15128.323753] Lustre: 435635:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.112@tcp: can't add NID 192.168.201.112@tcp130: rc = -75 [15128.347719] Lustre: 435635:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [15128.378607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15128.416280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15134.764528] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15147.649847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15147.742919] Lustre: 435596:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15147.773667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15147.776234] Lustre: Skipped 1 previous similar message [15147.848498] Lustre: lustre-MDT0001: new disk, initializing [15147.913358] Lustre: 436380:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.112@tcp: can't add NID 192.168.201.112@tcp130: rc = -75 [15147.917391] Lustre: 436380:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [15148.085634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15148.133701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15148.141604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15154.561432] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15154.744802] Lustre: 436629:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.112@tcp: can't add NID 192.168.201.112@tcp130: rc = -75 [15154.758967] Lustre: 436629:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [15157.804091] LNetError: 433039:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [15157.814818] LNetError: 433039:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 1 previous similar message [15157.825221] LNetError: Protocol error connecting to 192.168.201.112@tcp126 at host 192.168.201.112:988: is it running a compatible version of Lustre? [15160.864838] LNetError: 435609:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [15160.874260] LNetError: 435609:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 1 previous similar message [15160.883419] LNetError: Protocol error connecting to 192.168.201.112@tcp129 at host 192.168.201.112:988: is it running a compatible version of Lustre? [15160.891514] LNetError: Skipped 1 previous similar message [15162.092368] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15164.963463] LNetError: 435599:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [15164.975791] LNetError: 435599:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 2 previous similar messages [15164.985806] LNetError: Protocol error connecting to 192.168.201.112@tcp117 at host 192.168.201.112:988: is it running a compatible version of Lustre? [15164.996551] LNetError: Skipped 2 previous similar messages [15166.059779] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15172.616242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15172.924412] Lustre: lustre-OST0000: new disk, initializing [15172.928753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15172.997697] Lustre: 437349:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.112@tcp: can't add NID 192.168.201.112@tcp130: rc = -75 [15173.004584] Lustre: 437349:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [15173.060011] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15173.170826] LNetError: 433039:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [15173.187266] LNetError: 433039:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 4 previous similar messages [15173.197799] LNetError: Protocol error connecting to 192.168.201.112@tcp129 at host 192.168.201.112:988: is it running a compatible version of Lustre? [15173.208125] LNetError: Skipped 4 previous similar messages [15179.521464] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15180.346493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15180.358114] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15180.613342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15186.979885] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15189.541377] LNetError: 435611:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [15189.566967] LNetError: 435611:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 10 previous similar messages [15189.576504] LNetError: Protocol error connecting to 192.168.201.112@tcp117 at host 192.168.201.112:988: is it running a compatible version of Lustre? [15189.584296] LNetError: Skipped 10 previous similar messages [15196.134453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15196.136153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15196.136636] LustreError: Skipped 1 previous similar message [15196.138357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15196.144492] Lustre: Skipped 1 previous similar message [15196.155453] Lustre: Skipped 1 previous similar message [15200.039500] Lustre: server umount lustre-OST0000 complete [15204.320279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15204.323641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15204.346174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15206.368570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15206.371196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15206.406323] Lustre: Skipped 1 previous similar message [15210.564634] Lustre: server umount lustre-MDT0000 complete [15214.991518] LustreError: 435587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779987948 with bad export cookie 1279154062165623383 [15214.995552] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15215.004671] LustreError: 435587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15215.291414] Lustre: server umount lustre-MDT0001 complete [15222.307171] LNetError: 433038:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.112 [15222.321709] LNetError: 433038:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 11 previous similar messages [15222.337707] LNetError: Protocol error connecting to 192.168.201.112@tcp121 at host 192.168.201.112:988: is it running a compatible version of Lustre? [15222.351427] LNetError: Skipped 11 previous similar messages [15228.561203] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [15232.259786] Key type lgssc unregistered [15232.640191] LNet: 439127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15232.650529] LNetError: 439127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15232.654298] LNet: Removed LNI 192.168.201.112@tcp [15233.695215] LNet: Removed LNI 192.168.201.112@tcp100 [15233.701939] LNetError: 439127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15233.713649] LNetError: 439127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15233.720910] LNet: Removed LNI 192.168.201.112@tcp101 [15234.783453] LNet: Removed LNI 192.168.201.112@tcp102 [15234.785555] LNetError: 439127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15234.811902] LNetError: 439127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15234.841900] LNet: Removed LNI 192.168.201.112@tcp103 [15235.880120] LNet: Removed LNI 192.168.201.112@tcp104 [15236.895187] LNet: Removed LNI 192.168.201.112@tcp105 [15236.897436] LNetError: 439127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15236.903831] LNetError: 439127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15236.911798] LNet: Removed LNI 192.168.201.112@tcp106 [15237.983303] LNet: Removed LNI 192.168.201.112@tcp107 [15239.007494] LNet: Removed LNI 192.168.201.112@tcp108 [15240.031208] LNet: Removed LNI 192.168.201.112@tcp109 [15241.055941] LNet: Removed LNI 192.168.201.112@tcp110 [15241.059559] LNetError: 439127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15241.072624] LNetError: 439127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15241.094473] LNet: Removed LNI 192.168.201.112@tcp111 [15242.143235] LNet: Removed LNI 192.168.201.112@tcp112 [15243.167819] LNet: Removed LNI 192.168.201.112@tcp113 [15244.191397] LNet: Removed LNI 192.168.201.112@tcp114 [15245.215170] LNet: Removed LNI 192.168.201.112@tcp115 [15246.239241] LNet: Removed LNI 192.168.201.112@tcp116 [15247.264096] LNet: Removed LNI 192.168.201.112@tcp117 [15248.288066] LNet: Removed LNI 192.168.201.112@tcp118 [15249.311521] LNet: Removed LNI 192.168.201.112@tcp119 [15249.316057] LNetError: 439127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15249.338307] LNetError: 439127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15249.354229] LNet: Removed LNI 192.168.201.112@tcp120 [15250.400217] LNet: Removed LNI 192.168.201.112@tcp121 [15251.423653] LNet: Removed LNI 192.168.201.112@tcp122 [15252.447388] LNet: Removed LNI 192.168.201.112@tcp123 [15253.471230] LNet: Removed LNI 192.168.201.112@tcp124 [15254.495597] LNet: Removed LNI 192.168.201.112@tcp125 [15255.519558] LNet: Removed LNI 192.168.201.112@tcp126 [15256.543201] LNet: Removed LNI 192.168.201.112@tcp127 [15257.567986] LNet: Removed LNI 192.168.201.112@tcp128 [15258.591195] LNet: Removed LNI 192.168.201.112@tcp129 [15258.602406] LNet: Removed LNI 192.168.201.112@tcp130 [15258.608988] LNet: Removed LNI 192.168.201.112@tcp131 [15258.610902] LNet: Removed LNI 192.168.201.112@tcp132 [15258.617659] LNet: Removed LNI 192.168.201.112@tcp133 [15258.621643] LNet: Removed LNI 192.168.201.112@tcp134 [15258.636684] LNet: Removed LNI 192.168.201.112@tcp135 [15259.414394] Key type .llcrypt unregistered [15259.416202] Key type ._llcrypt unregistered [15281.885518] Key type ._llcrypt registered [15281.890978] Key type .llcrypt registered [15282.029335] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [15296.476096] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [15297.442263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15297.519877] alg: No test for adler32 (adler32-zlib) [15298.529757] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [15298.699708] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [15300.343192] Key type lgssc registered [15301.362813] Lustre: Echo OBD driver; http://www.lustre.org/ [15343.417571] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:07:54 (1779988074) [15357.675817] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [15359.708688] Key type lgssc unregistered [15359.988840] LNet: 443164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15359.993912] LNetError: 443164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15360.011421] LNet: Removed LNI 192.168.201.112@tcp [15360.551663] Key type .llcrypt unregistered [15360.558645] Key type ._llcrypt unregistered [15376.140700] Key type ._llcrypt registered [15376.142113] Key type .llcrypt registered [15376.289833] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [15377.643836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15377.750948] alg: No test for adler32 (adler32-zlib) [15378.899251] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [15379.212609] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [15380.879169] Key type lgssc registered [15381.762956] Lustre: Echo OBD driver; http://www.lustre.org/ [15390.232824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15390.286536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15391.628261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15391.694563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15391.753125] Lustre: lustre-MDT0000: new disk, initializing [15391.829185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15391.850878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15396.347769] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15406.494730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15406.600132] Lustre: 444361:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15406.660145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15406.664575] Lustre: Skipped 1 previous similar message [15406.737498] Lustre: lustre-MDT0001: new disk, initializing [15406.846884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15406.903692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15406.912942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15410.313557] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15417.287280] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15421.092423] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15427.313223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15427.515433] Lustre: lustre-OST0000: new disk, initializing [15427.518983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15427.596943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15433.361689] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15436.894410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15436.912464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15437.044803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15441.142257] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15446.651128] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing lnet_if_list [15454.367350] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15454.415332] LNet: Added LNI 192.168.201.112@tcp10 [8/256/0/180] [15461.534608] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15461.559546] LNet: Added LNI 192.168.201.112@tcp11 [8/256/0/180] [15467.216769] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15467.248761] LNet: Added LNI 192.168.201.112@tcp12 [8/256/0/180] [15472.971347] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15473.003699] LNet: Added LNI 192.168.201.112@tcp13 [8/256/0/180] [15477.944842] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15477.979850] LNet: Added LNI 192.168.201.112@tcp14 [8/256/0/180] [15484.673488] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15484.737723] LNet: Added LNI 192.168.201.112@tcp15 [8/256/0/180] [15491.725518] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15491.785035] LNet: Added LNI 192.168.201.112@tcp16 [8/256/0/180] [15498.511687] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15498.542801] LNet: Added LNI 192.168.201.112@tcp17 [8/256/0/180] [15504.989607] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15505.009500] LNet: Added LNI 192.168.201.112@tcp18 [8/256/0/180] [15511.274496] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15511.327616] LNet: Added LNI 192.168.201.112@tcp19 [8/256/0/180] [15515.771788] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15515.796573] LNet: Added LNI 192.168.201.112@tcp20 [8/256/0/180] [15521.168605] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15521.190306] LNet: Added LNI 192.168.201.112@tcp21 [8/256/0/180] [15525.401890] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15525.422693] LNet: Added LNI 192.168.201.112@tcp22 [8/256/0/180] [15529.400518] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15529.425307] LNet: Added LNI 192.168.201.112@tcp23 [8/256/0/180] [15534.505338] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15534.537795] LNet: Added LNI 192.168.201.112@tcp24 [8/256/0/180] [15540.552554] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15540.576579] LNet: Added LNI 192.168.201.112@tcp25 [8/256/0/180] [15545.994565] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15546.033210] LNet: Added LNI 192.168.201.112@tcp26 [8/256/0/180] [15552.255656] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15552.280650] LNet: Added LNI 192.168.201.112@tcp27 [8/256/0/180] [15559.346458] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15559.388628] LNet: Added LNI 192.168.201.112@tcp28 [8/256/0/180] [15566.266212] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15566.286376] LNet: Added LNI 192.168.201.112@tcp29 [8/256/0/180] [15573.105404] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15573.147239] LNet: Added LNI 192.168.201.112@tcp30 [8/256/0/180] [15579.276550] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15579.299860] LNet: Added LNI 192.168.201.112@tcp31 [8/256/0/180] [15585.623394] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15585.644802] LNet: Added LNI 192.168.201.112@tcp32 [8/256/0/180] [15591.056499] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15591.083909] LNet: Added LNI 192.168.201.112@tcp33 [8/256/0/180] [15597.824299] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15597.860826] LNet: Added LNI 192.168.201.112@tcp34 [8/256/0/180] [15605.291267] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15605.337377] LNet: Added LNI 192.168.201.112@tcp35 [8/256/0/180] [15615.612249] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15615.664758] LNet: Added LNI 192.168.201.112@tcp36 [8/256/0/180] [15625.855491] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15625.886886] LNet: Added LNI 192.168.201.112@tcp37 [8/256/0/180] [15632.533583] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15632.582410] LNet: Added LNI 192.168.201.112@tcp38 [8/256/0/180] [15639.669166] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15639.690038] LNet: Added LNI 192.168.201.112@tcp39 [8/256/0/180] [15646.443432] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15646.462458] LNet: Added LNI 192.168.201.112@tcp40 [8/256/0/180] [15653.457493] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15653.473477] LNet: Added LNI 192.168.201.112@tcp41 [8/256/0/180] [15661.182409] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15661.224436] LNet: Added LNI 192.168.201.112@tcp42 [8/256/0/180] [15669.610562] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15669.646989] LNet: Added LNI 192.168.201.112@tcp43 [8/256/0/180] [15680.057299] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15680.086558] LNet: Added LNI 192.168.201.112@tcp44 [8/256/0/180] [15689.873383] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15689.900789] LNet: Added LNI 192.168.201.112@tcp45 [8/256/0/180] [15698.898241] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15698.948344] LNet: Added LNI 192.168.201.112@tcp46 [8/256/0/180] [15706.978846] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15707.017132] LNet: Added LNI 192.168.201.112@tcp47 [8/256/0/180] [15717.998370] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15718.036621] LNet: Added LNI 192.168.201.112@tcp48 [8/256/0/180] [15730.127685] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15730.156084] LNet: Added LNI 192.168.201.112@tcp49 [8/256/0/180] [15741.858847] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15741.904167] LNet: Added LNI 192.168.201.112@tcp50 [8/256/0/180] [15764.961270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15764.972597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15764.990693] Lustre: Skipped 2 previous similar messages [15765.019789] Lustre: Skipped 2 previous similar messages [15770.098014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15770.101059] Lustre: Skipped 2 previous similar messages [15771.108528] Lustre: server umount lustre-MDT0000 complete [15775.128409] LustreError: 444351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779988508 with bad export cookie 13929411714855859158 [15775.149721] LustreError: 444351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15775.155144] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15775.201481] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15775.222388] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [15775.708456] Lustre: server umount lustre-MDT0001 complete [15799.851798] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [15810.348139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15820.611840] LustreError: 446098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15820.627430] LustreError: 446098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15820.724470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15825.864893] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15825.889581] LustreError: 446098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15829.987789] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15829.997596] LustreError: 446098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15835.105137] LustreError: 446098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15835.521367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15836.196382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15839.701604] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15840.323616] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15846.080963] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15850.867479] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15876.096290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15876.108632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15876.116354] Lustre: Skipped 1 previous similar message [15876.140907] Lustre: Skipped 1 previous similar message [15881.190843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15881.197333] Lustre: Skipped 1 previous similar message [15888.355444] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15888.536156] Lustre: server umount lustre-OST0000 complete [15892.478212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15892.488172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15892.509369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15892.512214] Lustre: Skipped 2 previous similar messages [15896.552584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15899.082782] Lustre: server umount lustre-MDT0000 complete [15901.664371] LustreError: 469634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15901.687818] LustreError: 469634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15904.704724] LustreError: 444353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779988637 with bad export cookie 13929411714855861776 [15904.725588] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15904.730859] LustreError: 444353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15905.004095] Lustre: server umount lustre-MDT0001 complete [15917.282901] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [15920.011775] Key type lgssc unregistered [15920.443683] LNet: 472194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15920.454804] LNetError: 472194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15920.465923] LNet: Removed LNI 192.168.201.112@tcp [15920.469383] LNet: Removed LNI 192.168.201.112@tcp10 [15920.471569] LNet: Removed LNI 192.168.201.112@tcp11 [15920.474375] LNet: Removed LNI 192.168.201.112@tcp12 [15920.476815] LNet: Removed LNI 192.168.201.112@tcp13 [15920.479716] LNet: Removed LNI 192.168.201.112@tcp14 [15920.482329] LNet: Removed LNI 192.168.201.112@tcp15 [15920.483955] LNet: Removed LNI 192.168.201.112@tcp16 [15920.485802] LNet: Removed LNI 192.168.201.112@tcp17 [15920.487921] LNet: Removed LNI 192.168.201.112@tcp18 [15920.489724] LNet: Removed LNI 192.168.201.112@tcp19 [15920.496900] LNet: Removed LNI 192.168.201.112@tcp20 [15920.499409] LNet: Removed LNI 192.168.201.112@tcp21 [15920.502337] LNet: Removed LNI 192.168.201.112@tcp22 [15920.506057] LNet: Removed LNI 192.168.201.112@tcp23 [15920.508718] LNet: Removed LNI 192.168.201.112@tcp24 [15920.511476] LNet: Removed LNI 192.168.201.112@tcp25 [15920.514151] LNet: Removed LNI 192.168.201.112@tcp26 [15920.517203] LNet: Removed LNI 192.168.201.112@tcp27 [15920.519755] LNet: Removed LNI 192.168.201.112@tcp28 [15920.526124] LNet: Removed LNI 192.168.201.112@tcp29 [15920.528727] LNet: Removed LNI 192.168.201.112@tcp30 [15920.534407] LNet: Removed LNI 192.168.201.112@tcp31 [15920.537361] LNet: Removed LNI 192.168.201.112@tcp32 [15920.539797] LNet: Removed LNI 192.168.201.112@tcp33 [15920.542515] LNet: Removed LNI 192.168.201.112@tcp34 [15920.545068] LNet: Removed LNI 192.168.201.112@tcp35 [15920.547381] LNet: Removed LNI 192.168.201.112@tcp36 [15920.549376] LNet: Removed LNI 192.168.201.112@tcp37 [15920.551786] LNet: Removed LNI 192.168.201.112@tcp38 [15920.554259] LNet: Removed LNI 192.168.201.112@tcp39 [15920.557507] LNet: Removed LNI 192.168.201.112@tcp40 [15920.561597] LNet: Removed LNI 192.168.201.112@tcp41 [15920.564522] LNet: Removed LNI 192.168.201.112@tcp42 [15920.569412] LNet: Removed LNI 192.168.201.112@tcp43 [15920.576707] LNet: Removed LNI 192.168.201.112@tcp44 [15920.580670] LNet: Removed LNI 192.168.201.112@tcp45 [15920.583267] LNet: Removed LNI 192.168.201.112@tcp46 [15920.585086] LNet: Removed LNI 192.168.201.112@tcp47 [15920.590655] LNet: Removed LNI 192.168.201.112@tcp48 [15920.592830] LNet: Removed LNI 192.168.201.112@tcp49 [15920.602782] LNet: Removed LNI 192.168.201.112@tcp50 [15921.433981] Key type .llcrypt unregistered [15921.442307] Key type ._llcrypt unregistered [15943.405393] Key type ._llcrypt registered [15943.409588] Key type .llcrypt registered [15943.545640] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [15944.791343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15944.861950] alg: No test for adler32 (adler32-zlib) [15945.871347] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [15946.128398] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [15947.855177] Key type lgssc registered [15948.874511] Lustre: Echo OBD driver; http://www.lustre.org/ [15956.864551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15956.886968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15958.274752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15962.704605] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15968.738768] LustreError: 473441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15968.763805] LustreError: 473441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15971.820445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15972.260373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15977.716851] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [15983.939732] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15989.461885] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15997.806147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15998.160587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16003.256200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [16005.304558] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [16013.099157] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16036.764816] loop0: detected capacity change from 0 to 204800000 [16037.222037] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16037.248431] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16037.274457] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16046.250285] loop0: detected capacity change from 0 to 204800000 [16046.522086] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16046.531178] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16046.541865] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16053.724401] loop0: detected capacity change from 0 to 204800000 [16061.587453] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16061.686331] Lustre: fs15246-OST0000: new disk, initializing [16061.694106] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16061.743876] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16068.566392] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [16079.165745] loop1: detected capacity change from 0 to 204800000 [16085.123623] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16085.237700] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16085.320712] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16085.413402] Lustre: fs15246-MDT0000: new disk, initializing [16085.628978] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16085.671630] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16085.813616] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16086.881045] Lustre: 477830:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000001968fcd7 x1866453409124608/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1779988835 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16091.440949] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [16095.982680] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16096.000488] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16096.040431] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16117.740948] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16117.764265] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16122.861250] LustreError: 473437:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16123.174988] Lustre: server umount fs15246-MDT0000 complete [16127.978732] LustreError: 473437:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16133.106222] LustreError: 473436:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16134.019928] Lustre: server umount fs15246-OST0000 complete [16148.452756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16148.462780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16148.484364] Lustre: Skipped 1 previous similar message [16148.496192] Lustre: Skipped 1 previous similar message [16153.572807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16153.595469] Lustre: Skipped 1 previous similar message [16154.041839] Lustre: server umount lustre-OST0000 complete [16161.760515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16161.775145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16161.803975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16163.812422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16165.283822] Lustre: server umount lustre-MDT0000 complete [16168.417170] LustreError: 473423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779988901 with bad export cookie 9878653361654543961 [16168.424861] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16168.427977] LustreError: 473423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16168.879399] Lustre: server umount lustre-MDT0001 complete [16180.502362] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [16183.996074] Key type lgssc unregistered [16184.356827] LNet: 480896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16184.378975] LNetError: 480896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16185.451444] LNet: Removed LNI 192.168.201.112@tcp [16186.358743] Key type .llcrypt unregistered [16186.360530] Key type ._llcrypt unregistered [16210.080840] Key type ._llcrypt registered [16210.083686] Key type .llcrypt registered [16210.218835] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [16212.084028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16212.275338] alg: No test for adler32 (adler32-zlib) [16213.733718] Lustre: Lustre: Build Version: 2.17.50_143_gf28c0c4 [16214.076228] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [16215.855234] Key type lgssc registered [16217.755880] Lustre: Echo OBD driver; http://www.lustre.org/ [16268.047436] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [16281.075199] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [16319.944060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16319.987065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16321.227913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16321.256708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16321.366072] Lustre: lustre-MDT0000: new disk, initializing [16321.430741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16321.454749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16326.280850] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [16336.470375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16336.626385] Lustre: 485313:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16336.659409] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16336.663727] Lustre: Skipped 1 previous similar message [16336.779516] Lustre: lustre-MDT0001: new disk, initializing [16336.932604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16337.005130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16337.019628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16342.035444] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [16352.253962] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16356.868362] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16366.501383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16366.736249] Lustre: lustre-OST0000: new disk, initializing [16366.740125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16366.831652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16372.868457] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [16373.879238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16373.903180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16374.038437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16382.457520] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16388.882290] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16389.254073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16395.510530] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16395.897066] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16399.849496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16399.850202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16399.854703] LustreError: Skipped 1 previous similar message [16399.868557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16399.869615] Lustre: Skipped 1 previous similar message [16399.906374] Lustre: Skipped 1 previous similar message [16404.975274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16404.988043] Lustre: Skipped 1 previous similar message [16410.102258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16410.105675] Lustre: Skipped 1 previous similar message [16413.152521] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16413.381825] Lustre: server umount lustre-OST0000 complete [16420.331256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16420.352654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16425.400930] Lustre: server umount lustre-MDT0000 complete [16429.903274] LustreError: 485307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779989162 with bad export cookie 5196756651080442265 [16429.905874] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16429.915604] LustreError: 485307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16430.285913] Lustre: server umount lustre-MDT0001 complete [16459.318901] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [16473.933524] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [16510.218534] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16237 sec ========= 13:27:21 (1779989241) [16512.695292] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:27:23 (1779989243) === [16517.129169] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:27:28 (1779989248) === [16560.736475] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [16564.142707] Key type lgssc unregistered [16564.453482] LNet: 493525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16564.473846] LNetError: 493525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16564.490198] LNet: Removed LNI 192.168.201.112@tcp [16565.334289] Key type .llcrypt unregistered [16565.337767] Key type ._llcrypt unregistered