[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496883045 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002368] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007701] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010115] LSM: Security Framework initializing [ 0.011040] Yama: becoming mindful. [ 0.012028] SELinux: Initializing. [ 0.013056] *** VALIDATE selinux *** [ 0.021269] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026209] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027129] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029094] *** VALIDATE tmpfs *** [ 0.031404] *** VALIDATE proc *** [ 0.032222] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.034248] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041021] debug: unmapping init [mem 0xffffffff96c59000-0xffffffff96c60fff] [ 0.043224] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044675] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.047010] ... generic registers: 4 [ 0.048043] ... value mask: 0000ffffffffffff [ 0.049010] ... max period: 00007fffffffffff [ 0.050010] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052331] rcu: Hierarchical SRCU implementation. [ 0.054448] smp: Bringing up secondary CPUs ... [ 0.055553] x86: Booting SMP configuration: [ 0.056021] .... node #0, CPUs: #1 #2 #3 [ 0.059440] smp: Brought up 1 node, 4 CPUs [ 0.060925] smpboot: Max logical packages: 1 [ 0.061011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137574] node 0 deferred pages initialised in 72ms [ 0.142095] devtmpfs: initialized [ 0.143302] x86/mm: Memory block size: 128MB [ 0.146810] gcov: version magic: 0x41383552 [ 0.150474] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157377] pinctrl core: initialized pinctrl subsystem [ 0.160155] [ 0.160993] ************************************************************* [ 0.164012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167009] ** ** [ 0.170013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.173011] ** ** [ 0.176011] ** This means that this kernel is built to expose internal ** [ 0.179012] ** IOMMU data structures, which may compromise security on ** [ 0.182016] ** your system. ** [ 0.184019] ** ** [ 0.187012] ** If you see this message and you are not debugging the ** [ 0.192014] ** kernel, report this immediately to your vendor! ** [ 0.195010] ** ** [ 0.197009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.199009] ************************************************************* [ 0.201536] NET: Registered protocol family 16 [ 0.204586] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.208055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.211106] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.214076] cpuidle: using governor menu [ 0.215795] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.219155] PCI: Using configuration type 1 for base access [ 0.221117] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.232190] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.235017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.239027] cryptd: max_cpu_qlen set to 1000 [ 0.241545] ACPI: Added _OSI(Module Device) [ 0.243009] ACPI: Added _OSI(Processor Device) [ 0.244010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.245010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250570] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.257399] ACPI: Interpreter enabled [ 0.259049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.260009] ACPI: Using IOAPIC for interrupt routing [ 0.262107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.266390] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.275826] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.279037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.285069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.290355] acpiphp: Slot [2] registered [ 0.293089] acpiphp: Slot [5] registered [ 0.294149] acpiphp: Slot [6] registered [ 0.295137] acpiphp: Slot [7] registered [ 0.297113] acpiphp: Slot [8] registered [ 0.298111] acpiphp: Slot [9] registered [ 0.300116] acpiphp: Slot [10] registered [ 0.301115] acpiphp: Slot [3] registered [ 0.302102] acpiphp: Slot [4] registered [ 0.304078] acpiphp: Slot [11] registered [ 0.305099] acpiphp: Slot [12] registered [ 0.307089] acpiphp: Slot [13] registered [ 0.308081] acpiphp: Slot [14] registered [ 0.310089] acpiphp: Slot [15] registered [ 0.311283] acpiphp: Slot [16] registered [ 0.313089] acpiphp: Slot [17] registered [ 0.315089] acpiphp: Slot [18] registered [ 0.317097] acpiphp: Slot [19] registered [ 0.318100] acpiphp: Slot [20] registered [ 0.320099] acpiphp: Slot [21] registered [ 0.322105] acpiphp: Slot [22] registered [ 0.324099] acpiphp: Slot [23] registered [ 0.325089] acpiphp: Slot [24] registered [ 0.327091] acpiphp: Slot [25] registered [ 0.328145] acpiphp: Slot [26] registered [ 0.330091] acpiphp: Slot [27] registered [ 0.332092] acpiphp: Slot [28] registered [ 0.333120] acpiphp: Slot [29] registered [ 0.335090] acpiphp: Slot [30] registered [ 0.336102] acpiphp: Slot [31] registered [ 0.337066] PCI host bridge to bus 0000:00 [ 0.339017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.341019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.343019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.345019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.348028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.351023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.353190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.356995] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.361273] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.374013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.379055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.382018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.385016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.389018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.392511] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.395835] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.399043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.401706] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.406012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.420960] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.425971] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.432000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.445014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.455013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.496015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.507499] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.514015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.522016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.538022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.549204] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.560020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.566019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.580019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.591226] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.597016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.604019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.619020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.630094] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.637020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.643022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.657025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.667054] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.683021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.692018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.715024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.737010] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.739527] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.742393] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.744385] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.746269] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.751133] iommu: Default domain type: Passthrough [ 0.753487] SCSI subsystem initialized [ 0.755148] ACPI: bus type USB registered [ 0.757181] usbcore: registered new interface driver usbfs [ 0.759090] usbcore: registered new interface driver hub [ 0.761108] usbcore: registered new device driver usb [ 0.763247] pps_core: LinuxPPS API ver. 1 registered [ 0.765014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.769080] PTP clock support registered [ 0.773105] EDAC MC: Ver: 3.0.0 [ 0.774535] PCI: Using ACPI for IRQ routing [ 0.776255] NetLabel: Initializing [ 0.777010] NetLabel: domain hash size = 128 [ 0.778010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.779199] NetLabel: unlabeled traffic allowed by default [ 0.782169] vgaarb: loaded [ 0.784425] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.786029] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.795924] clocksource: Switched to clocksource kvm-clock [ 0.915343] VFS: Disk quotas dquot_6.6.0 [ 0.916764] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.919309] *** VALIDATE ramfs *** [ 0.920598] *** VALIDATE hugetlbfs *** [ 0.922221] pnp: PnP ACPI init [ 0.925218] pnp: PnP ACPI: found 6 devices [ 0.940879] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.944171] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.946576] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.948554] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.951224] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.953480] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.956980] NET: Registered protocol family 2 [ 0.959751] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.964745] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.968804] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.974387] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.980081] TCP: Hash tables configured (established 65536 bind 65536) [ 0.983261] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.986611] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.990637] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.993548] NET: Registered protocol family 1 [ 0.996760] RPC: Registered named UNIX socket transport module. [ 0.999482] RPC: Registered udp transport module. [ 1.001446] RPC: Registered tcp transport module. [ 1.003374] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.005916] NET: Registered protocol family 44 [ 1.007780] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.010153] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.012466] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.015061] PCI: CLS 0 bytes, default 64 [ 1.017716] Unpacking initramfs... [ 2.579369] debug: unmapping init [mem 0xffffa0343cc54000-0xffffa0343ffbffff] [ 2.584246] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.586577] software IO TLB: mapped [mem 0x00000000a1800000-0x00000000a5800000] (64MB) [ 2.589934] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.122984] Initialise system trusted keyrings [ 3.124936] Key type blacklist registered [ 3.127135] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.137115] zbud: loaded [ 3.140828] *** VALIDATE nfs *** [ 3.142059] *** VALIDATE nfs4 *** [ 3.143514] pstore: using deflate compression [ 3.146579] Platform Keyring initialized [ 3.263917] NET: Registered protocol family 38 [ 3.265658] Key type asymmetric registered [ 3.267493] Asymmetric key parser 'x509' registered [ 3.269440] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.273062] io scheduler mq-deadline registered [ 3.274424] io scheduler kyber registered [ 3.276440] io scheduler bfq registered [ 3.278817] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.283160] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.285341] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.287661] ACPI: Power Button [PWRF] [ 3.294615] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.305191] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.327433] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.335564] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.351234] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.379143] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.411177] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.417362] Non-volatile memory driver v1.3 [ 3.420403] Linux agpgart interface v0.103 [ 3.459309] virtio_blk virtio1: [vda] 149792 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.463118] vda: detected capacity change from 0 to 76693504 [ 3.478913] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.481551] vdb: detected capacity change from 0 to 1073741824 [ 3.496322] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.500380] vdc: detected capacity change from 0 to 2621440000 [ 3.517820] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.520385] vdd: detected capacity change from 0 to 2621440000 [ 3.539418] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.542674] vde: detected capacity change from 0 to 4294967296 [ 3.559502] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.562614] vdf: detected capacity change from 0 to 4294967296 [ 3.574798] libphy: Fixed MDIO Bus: probed [ 3.580334] usbcore: registered new interface driver usbserial_generic [ 3.582736] usbserial: USB Serial support registered for generic [ 3.587069] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.591098] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.593130] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.595633] mousedev: PS/2 mouse device common for all mice [ 3.599385] rtc_cmos 00:05: RTC can wake from S4 [ 3.603609] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.611578] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.619641] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.624144] rtc_cmos 00:05: registered as rtc0 [ 3.626054] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.629294] intel_pstate: CPU model not supported [ 3.632761] hid: raw HID events driver (C) Jiri Kosina [ 3.634905] usbcore: registered new interface driver usbhid [ 3.637203] usbhid: USB HID core driver [ 3.638749] drop_monitor: Initializing network drop monitor service [ 3.641472] Initializing XFRM netlink socket [ 3.643723] NET: Registered protocol family 10 [ 3.646638] Segment Routing with IPv6 [ 3.649611] NET: Registered protocol family 17 [ 3.652460] mpls_gso: MPLS GSO support [ 3.659220] RAS: Correctable Errors collector initialized. [ 3.661894] AVX version of gcm_enc/dec engaged. [ 3.664660] AES CTR mode by8 optimization enabled [ 3.767287] sched_clock: Marking stable (3767238636, 0)->(4632757357, -865518721) [ 3.772337] registered taskstats version 1 [ 3.774529] Loading compiled-in X.509 certificates [ 3.777202] zswap: loaded using pool lzo/zbud [ 3.804875] Key type big_key registered [ 3.819128] Key type encrypted registered [ 3.820626] ima: No TPM chip found, activating TPM-bypass! [ 3.825296] ima: Allocated hash algorithm: sha1 [ 3.826911] ima: No architecture policies found [ 3.829441] evm: Initialising EVM extended attributes: [ 3.832074] evm: security.selinux [ 3.833546] evm: security.ima [ 3.834960] evm: security.capability [ 3.836170] evm: HMAC attrs: 0x1 [ 3.839413] rtc_cmos 00:05: setting system clock to 2026-09-03 20:46:44 UTC (1788468404) [ 3.845988] debug: unmapping init [mem 0xffffffff97c03000-0xffffffff97dfffff] [ 3.850568] debug: unmapping init [mem 0xffffffff96982000-0xffffffff96c58fff] [ 3.860338] Write protecting the kernel read-only data: 28672k [ 3.864206] debug: unmapping init [mem 0xffffffff95003000-0xffffffff951fffff] [ 3.867582] debug: unmapping init [mem 0xffffffff95914000-0xffffffff959fffff] [ 3.910734] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.918697] systemd[1]: Detected virtualization kvm. [ 3.921243] systemd[1]: Detected architecture x86-64. [ 3.923608] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.949704] systemd[1]: No hostname configured. [ 3.957638] systemd[1]: Set hostname to . [ 3.960758] random: systemd: uninitialized urandom read (16 bytes read) [ 3.963205] systemd[1]: Initializing machine ID from random generator. [ 4.103357] random: systemd: uninitialized urandom read (16 bytes read) [ 4.106505] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.111613] random: systemd: uninitialized urandom read (16 bytes read) [ 4.114651] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.118356] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.764354] device-mapper: uevent: version 1.0.3 [ 4.766876] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.525184] virtio_net virtio0 ens2: renamed from eth0 [ 5.535349] random: fast init done [ 5.582228] scsi host0: ata_piix [ 5.600916] scsi host1: ata_piix [ 5.603205] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.605685] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.057576] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.348129] random: crng init done [ 10.350292] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.891350] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.203661] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.486575] SELinux: Disabled at runtime. [ 12.547186] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.556876] systemd[1]: Detected virtualization kvm. [ 12.558867] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.177916] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.181726] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.187403] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.191913] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.196180] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.204997] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.218045] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ 13.289897] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.684409] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.073853] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.080712] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.198312] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.214289] EDAC sbridge: Ver: 1.1.2 [ 16.008871] Key type dns_resolver registered [ 16.340554] NFS: Registering the id_resolver key type [ 16.343057] Key type id_resolver registered [ 16.344996] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg634-server login: [ 61.920024] hrtimer: interrupt took 5015612 ns [ 62.502568] libcfs: loading out-of-tree module taints kernel. [ 62.600458] Key type ._llcrypt registered [ 62.602329] Key type .llcrypt registered [ 62.722854] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 81.148958] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 82.615762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.625547] alg: No test for adler32 (adler32-zlib) [ 84.015697] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 84.723428] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 86.466792] Key type lgssc registered [ 87.859521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 106.477114] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 147.211731] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 161.728811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.746509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.021617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.054249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.179351] Lustre: lustre-MDT0000: new disk, initializing [ 163.399484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.414342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.806557] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.244800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.328862] Lustre: 6501:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 183.399381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 183.412439] Lustre: Skipped 1 previous similar message [ 183.563170] Lustre: lustre-MDT0001: new disk, initializing [ 183.623333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.652788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 183.666232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 187.387907] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.320819] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.650662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.880540] Lustre: lustre-OST0000: new disk, initializing [ 201.884259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 201.888333] Lustre: 8438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.935354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.194921] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.907024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.914362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 208.001607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 221.463863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.606880] Lustre: lustre-OST0001: new disk, initializing [ 221.613588] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 221.627909] Lustre: 9513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.724933] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.992567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.000379] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 228.071785] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 228.105683] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.888623] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 248.355504] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 254.008941] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing check_logdir /tmp/testlogs/ [ 259.766657] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing yml_node [ 264.316319] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 267.346802] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 269.531244] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 270.915666] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 16:51:09 EDT 2026 [ 284.656675] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 285.969895] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.536950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 301.545833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.569586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.121043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.121146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.140930] Lustre: Skipped 1 previous similar message [ 305.157285] Lustre: Skipped 3 previous similar messages [ 306.492719] Lustre: server umount lustre-MDT0000 complete [ 309.134042] LustreError: 6491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468709 with bad export cookie 14448579393786601867 [ 309.134462] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 309.152891] LustreError: 6491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 309.165216] Lustre: 12377:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 310.240846] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.240986] Lustre: 12380:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 310.242452] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.249579] Lustre: Skipped 2 previous similar messages [ 315.362164] LustreError: 9320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 315.362613] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 315.366115] Lustre: 12377:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 315.393064] LustreError: 9320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 315.417418] Lustre: Skipped 2 previous similar messages [ 315.495308] Lustre: server umount lustre-MDT0001 complete [ 318.745442] Lustre: server umount lustre-OST0000 complete [ 320.482838] Lustre: 12380:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 322.065308] Lustre: server umount lustre-OST0001 complete [ 327.129251] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 334.298892] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 368.395405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 368.611504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 368.632657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 368.704502] Lustre: lustre-MDT0000: new disk, initializing [ 368.769047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.792304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 372.934184] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 382.638225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 382.762887] Lustre: 15071:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 382.891774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 382.896409] Lustre: Skipped 1 previous similar message [ 382.994298] Lustre: lustre-MDT0001: new disk, initializing [ 383.067105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 383.105929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 383.113338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 387.330275] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 396.480959] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 401.414478] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 408.069512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 408.510183] Lustre: lustre-OST0000: new disk, initializing [ 408.513482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 408.520902] Lustre: 17040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 408.670627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.134346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 410.150619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 410.279812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 415.819538] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 426.553284] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 430.757912] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 430.956103] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 435.028814] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 435.207706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 440.802336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 440.806808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.833967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 451.040173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 451.311270] Lustre: server umount lustre-OST0000 complete [ 454.639470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 454.657310] LustreError: Skipped 1 previous similar message [ 454.664261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.673618] Lustre: Skipped 1 previous similar message [ 454.677661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 454.682291] Lustre: Skipped 4 previous similar messages [ 460.797847] Lustre: server umount lustre-MDT0000 complete [ 461.793875] LustreError: 15078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 461.809115] LustreError: 15078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 464.936649] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468865 with bad export cookie 14448579393786603491 [ 464.942678] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 464.945840] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 464.961891] Lustre: 18650:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 465.193586] Lustre: server umount lustre-MDT0001 complete [ 469.656553] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:54:27 (1788468867) [ 473.390051] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 476.239910] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:54:33 (1788468873) [ 485.469956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.128031] LustreError: 19455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.246421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.532935] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 491.490396] LustreError: 19456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.587109] LustreError: 19455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.650302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.175690] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 511.495673] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 517.578936] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 523.908086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 529.913378] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 537.531360] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 556.011048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 556.013730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.020716] Lustre: Skipped 1 previous similar message [ 556.029263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 556.034132] Lustre: Skipped 2 previous similar messages [ 561.563562] Lustre: server umount lustre-OST0000 complete [ 565.730468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 565.739355] LustreError: Skipped 1 previous similar message [ 571.547864] Lustre: server umount lustre-MDT0000 complete [ 574.438173] LustreError: 19456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.454345] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.474759] LustreError: 19456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 574.500202] Lustre: 22323:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 575.965669] Lustre: server umount lustre-MDT0001 complete [ 587.468947] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 590.092540] Key type lgssc unregistered [ 590.380565] LNet: 23012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.388252] LNetError: 23012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 591.470027] LNet: Removed LNI 192.168.206.134@tcp [ 592.505303] Key type .llcrypt unregistered [ 592.508424] Key type ._llcrypt unregistered [ 615.316550] Key type ._llcrypt registered [ 615.322904] Key type .llcrypt registered [ 615.436767] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 616.301299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 616.702252] alg: No test for adler32 (adler32-zlib) [ 618.238493] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 618.652724] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 620.289250] Key type lgssc registered [ 622.099664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 634.542097] loop: module loaded [ 634.638407] loop0: detected capacity change from 0 to 204800000 [ 634.988307] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 635.005460] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 635.690984] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 642.617849] loop0: detected capacity change from 0 to 204800000 [ 642.765899] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 642.772693] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 642.780381] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 648.121379] loop0: detected capacity change from 0 to 204800000 [ 656.549511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 656.601860] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.227931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 658.319730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 658.414872] Lustre: lustre-MDT0000: new disk, initializing [ 658.652673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 658.686541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 666.085983] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 678.749889] loop1: detected capacity change from 0 to 204800000 [ 686.857969] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 687.282811] Lustre: lustre-OST07c6: new disk, initializing [ 687.285944] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 687.298651] Lustre: Skipped 1 previous similar message [ 687.324547] Lustre: 26209:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 687.516433] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 694.336209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 694.349942] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 694.393493] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 695.995232] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 701.222420] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 701.234528] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 701.240150] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 701.244212] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 701.250539] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 701.256140] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 719.851131] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 719.859915] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 719.872405] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 719.884328] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 719.893572] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 719.898677] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.218898] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 725.231880] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 725.234845] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 725.237691] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.243198] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.247547] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.251661] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 725.257199] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.261896] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 725.266561] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.270106] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.274537] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.544095] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 728.567435] Lustre: 25191:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 728.573372] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 728.592535] Lustre: 25191:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.614802] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 728.630921] Lustre: 25191:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.640948] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 728.649888] Lustre: 25191:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.671386] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 728.684196] Lustre: 25191:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.691778] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 728.710244] Lustre: 25191:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.041657] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 734.064789] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 734.083409] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 734.100311] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.109683] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 734.127731] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.134748] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 734.149948] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.156177] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 734.164880] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.174989] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 734.193890] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.380391] Lustre: 26897:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 742.402719] Lustre: 26897:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 742.415334] Lustre: 26897:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 742.434396] Lustre: 26897:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 742.441860] Lustre: 26897:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 742.451176] Lustre: 26897:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 742.465871] Lustre: 26897:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 742.474811] Lustre: 26897:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 742.482768] Lustre: 26897:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 742.502882] Lustre: 26897:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 742.524682] Lustre: 26897:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 742.544319] Lustre: 26897:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 796.641642] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 796.651332] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.673529] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 800.675475] Lustre: server umount lustre-OST07c6 complete [ 809.297362] Lustre: server umount lustre-MDT0000 complete [ 824.690789] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 827.224617] Key type lgssc unregistered [ 827.564556] LNet: 28769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.570438] LNetError: 28769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.596955] LNet: Removed LNI 192.168.206.134@tcp [ 828.365582] Key type .llcrypt unregistered [ 828.369168] Key type ._llcrypt unregistered [ 846.921333] Key type ._llcrypt registered [ 846.927917] Key type .llcrypt registered [ 847.037901] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 848.284215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 848.304969] alg: No test for adler32 (adler32-zlib) [ 849.401469] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 849.672874] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 851.387317] Key type lgssc registered [ 852.697279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 860.083514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 860.114149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.660163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.357143] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 871.910939] LustreError: 30048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 871.929039] LustreError: 30048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 875.291688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.760632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 881.026471] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 890.138945] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 897.593158] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 904.297549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 904.700366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.758801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 910.984182] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 918.564951] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 936.420251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 936.423663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 936.431119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 936.433622] Lustre: Skipped 1 previous similar message [ 936.442578] Lustre: Skipped 1 previous similar message [ 940.211871] Lustre: server umount lustre-OST0000 complete [ 946.658239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.661491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 946.675970] Lustre: Skipped 1 previous similar message [ 950.528363] Lustre: server umount lustre-MDT0000 complete [ 951.784306] LustreError: 30049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.798340] LustreError: 30049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 954.188080] LustreError: 30016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469354 with bad export cookie 421654670860246754 [ 954.194230] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.588943] Lustre: server umount lustre-MDT0001 complete [ 967.166804] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 971.100331] Key type lgssc unregistered [ 971.510348] LNet: 33783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.520563] LNetError: 33783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.582679] LNet: Removed LNI 192.168.206.134@tcp [ 973.751130] Key type .llcrypt unregistered [ 973.753177] Key type ._llcrypt unregistered [ 991.552434] Key type ._llcrypt registered [ 991.554546] Key type .llcrypt registered [ 991.649340] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 992.622854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 992.662460] alg: No test for adler32 (adler32-zlib) [ 993.670363] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 993.963569] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 995.688571] Key type lgssc registered [ 996.991616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1004.784349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1004.810660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.334415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.138167] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1015.788746] LustreError: 35050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1015.808096] LustreError: 35050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1019.908049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.343907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1025.098661] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1031.505629] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1037.151319] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1043.322243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1043.819808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.470127] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1051.017545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1060.546333] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1075.681386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1075.688150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.701540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1076.706288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1076.715390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1076.721337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.739383] Lustre: Skipped 1 previous similar message [ 1078.394864] Lustre: server umount lustre-OST0000 complete [ 1086.949161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.958937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1086.962111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1086.982115] Lustre: Skipped 1 previous similar message [ 1089.531485] Lustre: server umount lustre-MDT0000 complete [ 1092.068301] LustreError: 35051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1092.094091] LustreError: 35051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1094.286974] LustreError: 35031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469494 with bad export cookie 17201756949058925900 [ 1094.289920] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1094.301669] LustreError: 35031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1094.947381] Lustre: server umount lustre-MDT0001 complete [ 1107.914187] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 1110.848577] Key type lgssc unregistered [ 1111.231147] LNet: 38586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.242904] LNetError: 38586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.257840] LNet: Removed LNI 192.168.206.134@tcp [ 1112.560410] Key type .llcrypt unregistered [ 1112.565111] Key type ._llcrypt unregistered [ 1137.446603] Key type ._llcrypt registered [ 1137.449291] Key type .llcrypt registered [ 1137.532803] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 1138.410627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1138.450350] alg: No test for adler32 (adler32-zlib) [ 1139.455693] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 1139.761989] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 1141.552154] Key type lgssc registered [ 1142.679192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1150.467048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1150.483168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.032815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.337022] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1162.225673] LustreError: 40005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.282669] LustreError: 40005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1165.307323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1166.041062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1170.976897] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1179.030699] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1184.693336] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1191.203054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1191.456976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.584078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1197.908585] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1205.360793] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1215.972959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.980092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1215.994114] Lustre: Skipped 2 previous similar messages [ 1216.012621] Lustre: Skipped 2 previous similar messages [ 1217.554551] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.34@tcp (stopping) [ 1220.615457] Lustre: server umount lustre-MDT0000 complete [ 1221.089297] LustreError: 40005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.631979] LustreError: 40001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.659444] LustreError: 40001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1224.170277] LustreError: 42583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469624 with bad export cookie 5367417499684369904 [ 1224.173400] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1224.190264] LustreError: 42583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1226.209981] LustreError: 40000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1226.227773] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1226.255789] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1229.220059] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1229.225610] Lustre: Skipped 1 previous similar message [ 1230.756356] Lustre: server umount lustre-MDT0001 complete [ 1234.337659] Lustre: 42787:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1239.520921] Lustre: 42787:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1244.644709] Lustre: 42790:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1250.784988] Lustre: 42787:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1250.804342] Lustre: 42787:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [ 1251.562356] Lustre: server umount lustre-OST0000 complete [ 1269.037213] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 1273.357874] Key type lgssc unregistered [ 1273.797264] LNet: 43870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.809214] LNetError: 43870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1273.835116] LNet: Removed LNI 192.168.206.134@tcp [ 1275.668743] Key type .llcrypt unregistered [ 1275.676525] Key type ._llcrypt unregistered [ 1301.157785] Key type ._llcrypt registered [ 1301.159789] Key type .llcrypt registered [ 1301.266860] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 1302.172497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1302.196775] alg: No test for adler32 (adler32-zlib) [ 1303.230925] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 1303.421977] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 1305.048133] Key type lgssc registered [ 1306.067223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1311.794096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1311.815598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1313.114733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.551692] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1322.466744] LustreError: 45262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1322.490828] LustreError: 45262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1326.491795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1326.860629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1332.007903] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1340.057678] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1348.003875] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1357.527644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1357.775627] LustreError: 47086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.327724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.441707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1364.699624] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1372.137289] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1381.859544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.867472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1381.873805] Lustre: Skipped 1 previous similar message [ 1381.875181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1381.916694] Lustre: Skipped 1 previous similar message [ 1385.034791] Lustre: lustre-OST0000: Not available for connect from 192.168.206.34@tcp (stopping) [ 1386.180118] Lustre: server umount lustre-OST0000 complete [ 1407.464842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.480314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1407.486757] Lustre: Skipped 1 previous similar message [ 1412.585838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1412.604804] Lustre: Skipped 2 previous similar messages [ 1414.279173] Lustre: server umount lustre-MDT0000 complete [ 1417.697774] LustreError: 47616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1417.745655] LustreError: 47616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1420.348776] LustreError: 45244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469821 with bad export cookie 3221245819704970988 [ 1420.351467] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1420.369927] LustreError: 45244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1426.857242] Lustre: server umount lustre-MDT0001 complete [ 1438.232601] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 1441.344526] Key type lgssc unregistered [ 1441.557218] LNet: 48972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.566951] LNetError: 48972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.577362] LNet: Removed LNI 192.168.206.134@tcp [ 1442.586799] Key type .llcrypt unregistered [ 1442.592697] Key type ._llcrypt unregistered [ 1472.323463] Key type ._llcrypt registered [ 1472.325714] Key type .llcrypt registered [ 1472.409844] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 1473.304407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1473.561519] alg: No test for adler32 (adler32-zlib) [ 1474.662931] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 1475.044624] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 1476.708277] Key type lgssc registered [ 1478.307931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1487.089513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1487.126257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1488.678449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1493.250627] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1499.106876] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1499.154276] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1503.743473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1503.980413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1509.058577] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1517.964708] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1524.769761] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1532.522247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1532.898884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1533.938528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1538.939392] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1546.788304] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1553.216311] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1558.187205] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1571.176261] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0317h12m48s [ 1572.953291] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1575.905151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1575.926404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1575.954544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1577.118791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1577.134261] Lustre: Skipped 4 previous similar messages [ 1580.077946] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.34@tcp (stopping) [ 1581.976868] Lustre: server umount lustre-MDT0000 complete [ 1583.600954] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1583.647317] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1586.574715] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1587.699227] LustreError: 50384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1589.736122] LustreError: 50390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1589.762179] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1589.766414] LustreError: 50390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1589.819178] Lustre: 53514:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1593.911336] LustreError: 50384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1594.853921] Lustre: 53515:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1595.901773] Lustre: 53516:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1596.189945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.607061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1596.660545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1597.646990] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1602.246899] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1606.089391] Lustre: DEBUG MARKER: Wait for df (36716) ... [ 1607.782749] Lustre: DEBUG MARKER: done [ 1613.290767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1613.301701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1613.314444] Lustre: Skipped 3 previous similar messages [ 1613.327610] Lustre: Skipped 1 previous similar message [ 1618.427570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1618.451100] Lustre: Skipped 2 previous similar messages [ 1627.111103] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1627.281433] Lustre: server umount lustre-OST0000 complete [ 1633.764198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.796992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1633.799623] Lustre: Skipped 2 previous similar messages [ 1639.180480] Lustre: server umount lustre-MDT0000 complete [ 1640.933320] LustreError: 53076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.946175] LustreError: 53076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1643.021812] LustreError: 50370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470043 with bad export cookie 14977887583295882784 [ 1643.029985] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1643.032961] LustreError: 50370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1643.268330] Lustre: server umount lustre-MDT0001 complete [ 1654.976922] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 1657.705282] Key type lgssc unregistered [ 1657.950700] LNet: 55318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.956484] LNetError: 55318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1657.969586] LNet: Removed LNI 192.168.206.134@tcp [ 1658.875676] Key type .llcrypt unregistered [ 1658.877732] Key type ._llcrypt unregistered [ 1701.255244] Key type ._llcrypt registered [ 1701.260286] Key type .llcrypt registered [ 1701.429443] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 1702.273369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1702.362643] alg: No test for adler32 (adler32-zlib) [ 1703.514570] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 1703.831995] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 1705.560160] Key type lgssc registered [ 1706.535796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1717.474966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1717.514387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.807870] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1718.833672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1719.069187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.349729] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1734.749398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.903203] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1734.995479] Lustre: 57528:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1735.549742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1741.975851] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1752.782488] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1761.511177] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1770.926865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1771.157172] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1771.487293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.272691] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 1780.151037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1788.018174] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1798.073115] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0317h16m35s [ 1800.928830] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1813.976630] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1818.180634] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1818.238553] LustreError: 57536:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1838.606868] Lustre: lustre-MDT0000: Client 52bacc98-3eb2-4dc0-8c12-57547cc03045 (at 192.168.206.34@tcp) reconnecting [ 1858.256547] LustreError: 57536:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1861.527791] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1865.806341] Lustre: DEBUG MARKER: done [ 1877.484303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1877.484681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.498819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.501220] Lustre: Skipped 1 previous similar message [ 1877.521702] Lustre: Skipped 1 previous similar message [ 1880.998030] Lustre: server umount lustre-OST0000 complete [ 1887.714682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1887.724184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1887.750634] Lustre: Skipped 1 previous similar message [ 1887.770089] Lustre: Skipped 1 previous similar message [ 1888.818846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1888.826065] Lustre: Skipped 1 previous similar message [ 1893.002697] Lustre: server umount lustre-MDT0000 complete [ 1898.306090] LustreError: 57520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470298 with bad export cookie 13498696940985773231 [ 1898.309838] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1898.319098] LustreError: 57520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1898.358080] Lustre: 61300:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1898.943490] Lustre: server umount lustre-MDT0001 complete [ 1910.466928] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 1913.818153] Key type lgssc unregistered [ 1914.165981] LNet: 61871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.174340] LNetError: 61871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.193855] LNet: Removed LNI 192.168.206.134@tcp [ 1915.138092] Key type .llcrypt unregistered [ 1915.140535] Key type ._llcrypt unregistered [ 1978.791435] Key type ._llcrypt registered [ 1978.793918] Key type .llcrypt registered [ 1978.902522] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 1980.270806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.354464] alg: No test for adler32 (adler32-zlib) [ 1981.902931] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 1982.279136] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 1983.952232] Key type lgssc registered [ 1985.496487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1990.125964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1990.144519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.509811] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1991.567405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1991.979229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.536106] Lustre: Failing over lustre-MDT0000 [ 1994.047797] Lustre: server umount lustre-MDT0000 complete [ 2006.250436] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:20:03 (1788470403) [ 2015.670585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.586688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.534726] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2030.731880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.834026] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2030.880307] Lustre: 64671:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2031.173263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2035.332044] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2043.273964] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2049.554601] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2057.037131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2057.218328] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2057.538088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.643175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2064.684504] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2071.813513] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2078.438601] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2087.395805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2087.409129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.422594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.445275] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.447766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.469618] Lustre: Skipped 1 previous similar message [ 2092.614204] Lustre: server umount lustre-MDT0000 complete [ 2097.187775] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470497 with bad export cookie 13642000465683982224 [ 2097.207649] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2097.218362] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2097.730859] Lustre: server umount lustre-MDT0001 complete [ 2102.494876] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2106.852576] Lustre: 67597:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2107.396456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2111.974171] Lustre: 67601:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2116.070738] Lustre: 63588:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470500/real 1788470500] req@00000000eae21d64 x1875347119319168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470516 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2116.110962] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.144205] Lustre: Skipped 1 previous similar message [ 2116.156366] Lustre: 67597:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2117.087202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.502657] LustreError: 66518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.518211] LustreError: 66518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2117.597875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.690717] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2118.624299] Lustre: 63589:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470503/real 1788470503] req@000000009204ac33 x1875347119319296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470519 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2119.720163] LustreError: 66519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.723586] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2120.736844] Lustre: 63590:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470505/real 1788470505] req@0000000077c1ee92 x1875347119319424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470521 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2121.765876] LustreError: 66518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.667199] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2123.745873] Lustre: 63591:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470507/real 1788470507] req@0000000077121829 x1875347119319680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2125.878925] LustreError: 68441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.903938] LustreError: 68441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2127.901150] Lustre: 63588:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470512/real 1788470512] req@000000001e6bc41f x1875347119319936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2130.914825] LustreError: 68442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.951308] LustreError: 68442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2132.844191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.451156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2138.615763] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2139.166325] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2148.157385] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2154.101760] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2160.835858] Lustre: server umount lustre-MDT0000 complete [ 2164.195883] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.196932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2164.198533] LustreError: 69087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.221321] Lustre: Skipped 1 previous similar message [ 2164.428540] LustreError: 66524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470565 with bad export cookie 13642000465683984527 [ 2164.446825] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2164.818609] Lustre: server umount lustre-MDT0001 complete [ 2170.894878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2174.439704] Lustre: 70472:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2179.558147] Lustre: 70475:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2181.888385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2182.549773] LustreError: 66519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.553337] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.565773] LustreError: 66519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2182.584602] Lustre: Skipped 1 previous similar message [ 2182.627982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2182.688503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2183.670220] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2184.673111] Lustre: 63589:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470569/real 1788470569] req@0000000054c9cfae x1875347119348096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470585 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2187.347132] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2197.234140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.115557] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2203.541719] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2212.870574] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2219.090569] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2228.709026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.719288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2228.725837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.725845] Lustre: Skipped 1 previous similar message [ 2228.730416] Lustre: Skipped 1 previous similar message [ 2232.968078] Lustre: server umount lustre-MDT0000 complete [ 2233.831197] LustreError: 71126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.847361] LustreError: 71126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2237.351917] LustreError: 66524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470638 with bad export cookie 13642000465683985948 [ 2237.356317] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2237.359636] LustreError: 66524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2237.645970] Lustre: server umount lustre-MDT0001 complete [ 2245.237037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2248.097509] Lustre: 73107:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2248.642268] Lustre: DEBUG MARKER: files compared the same [ 2260.668692] Lustre: server umount lustre-OST0000 complete [ 2279.021362] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 2282.008220] Key type lgssc unregistered [ 2282.272635] LNet: 74408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.282569] LNetError: 74408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2282.310597] LNet: Removed LNI 192.168.206.134@tcp [ 2283.174789] Key type .llcrypt unregistered [ 2283.178571] Key type ._llcrypt unregistered [ 2307.722694] Key type ._llcrypt registered [ 2307.734073] Key type .llcrypt registered [ 2307.879181] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 2309.288746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2309.336979] alg: No test for adler32 (adler32-zlib) [ 2310.478526] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 2310.766978] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 2312.456553] Key type lgssc registered [ 2313.809684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2322.845550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2322.866357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.520632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.425442] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug +malloc all [ 2334.694993] LustreError: 75821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.716825] LustreError: 75821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2337.616639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.968384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2342.203824] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug +malloc all [ 2350.149638] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2356.346183] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2363.509586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2363.781095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.962606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2370.491946] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug +malloc all [ 2378.346875] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2390.501729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2390.512165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2390.540387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2393.570739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2393.572116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2393.582609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.319327] Lustre: server umount lustre-OST0000 complete [ 2403.815776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2403.836378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2403.847050] Lustre: Skipped 2 previous similar messages [ 2408.932728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2408.947937] Lustre: Skipped 1 previous similar message [ 2414.050323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2414.062073] Lustre: Skipped 1 previous similar message [ 2415.072274] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2415.221789] Lustre: server umount lustre-MDT0000 complete [ 2418.990726] LustreError: 75803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470819 with bad export cookie 16467007501186430101 [ 2418.997215] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2418.999364] LustreError: 75803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2419.386248] Lustre: server umount lustre-MDT0001 complete [ 2431.778271] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 2435.043347] Key type lgssc unregistered [ 2435.481603] LNet: 79357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2435.502806] LNetError: 79357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2435.522118] LNet: Removed LNI 192.168.206.134@tcp [ 2436.311289] Key type .llcrypt unregistered [ 2436.312785] Key type ._llcrypt unregistered [ 2463.005249] Key type ._llcrypt registered [ 2463.006972] Key type .llcrypt registered [ 2463.175891] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 2464.486030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2464.565826] alg: No test for adler32 (adler32-zlib) [ 2465.745162] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 2466.137663] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 2467.816162] Key type lgssc registered [ 2469.245525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2478.003391] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2478.042716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2479.244089] Lustre: 80759:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2494.816681] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2499.936324] LustreError: 80747:0:(mgc_request.c:317:do_config_log_add()) MGC192.168.206.134@tcp: failed processing log, type 4: rc = -110 [ 2504.673503] Lustre: 80761:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2515.379885] Lustre: 80759:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2520.551949] Lustre: 80761:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2525.664205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2525.682269] Lustre: 80759:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2532.827650] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2535.906025] Lustre: 80759:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2535.934679] Lustre: 80759:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [ 2541.818167] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2550.525489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.601567] Lustre: *** cfs_fail_loc=706, val=0*** [ 2550.858839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2550.931721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2555.879568] LustreError: 80776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.899910] LustreError: 80776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2557.911640] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2560.994187] LustreError: 80777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.032112] LustreError: 80777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2564.086422] LustreError: 80776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.115038] LustreError: 80776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2569.191605] LustreError: 81837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.220972] LustreError: 81837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2569.342615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.911972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2575.021927] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2583.133691] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2589.330143] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2591.715244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2591.727624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.745480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2595.300531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2595.315135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.341615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2597.795291] Lustre: server umount lustre-OST0000 complete [ 2605.539130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2605.550102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.576268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2605.586524] Lustre: Skipped 1 previous similar message [ 2611.171398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2611.179027] Lustre: Skipped 3 previous similar messages [ 2616.288176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2616.297234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2616.313382] Lustre: Skipped 1 previous similar message [ 2616.523250] Lustre: server umount lustre-MDT0000 complete [ 2620.111169] LustreError: 80753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471020 with bad export cookie 6746834215135010787 [ 2620.116146] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2620.124596] LustreError: 80753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2620.477169] Lustre: server umount lustre-MDT0001 complete [ 2632.099078] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 2635.219208] Key type lgssc unregistered [ 2635.529289] LNet: 84370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2635.539592] LNetError: 84370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2635.559142] LNet: Removed LNI 192.168.206.134@tcp [ 2636.537237] Key type .llcrypt unregistered [ 2636.539972] Key type ._llcrypt unregistered [ 2662.136876] Key type ._llcrypt registered [ 2662.140427] Key type .llcrypt registered [ 2662.225432] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 2663.466399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2663.488650] alg: No test for adler32 (adler32-zlib) [ 2664.608307] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 2664.923395] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 2666.632166] Key type lgssc registered [ 2667.695363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2675.855895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2675.879983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.023556] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2688.272853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2688.696043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2693.411519] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2694.114646] LustreError: 86479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.144741] LustreError: 86479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2699.236130] LustreError: 86482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.259906] LustreError: 86482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2701.953468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2702.327400] LustreError: 86479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.346032] LustreError: 86479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2702.364794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2707.433746] LustreError: 86483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.010951] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2712.545903] LustreError: 86482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2712.574562] LustreError: 86482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2717.812650] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2717.832981] Lustre: MGS: Client bad1809d-1c77-49f7-9f73-587bcfe21aff (at 0@lo) reconnecting [ 2717.858651] LustreError: 86493:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2718.250621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2718.349646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2722.862803] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2738.663582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2738.676303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.679037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2738.714557] Lustre: Skipped 1 previous similar message [ 2742.894364] Lustre: server umount lustre-OST0000 complete [ 2748.898335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.902559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2748.921513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2748.947025] Lustre: Skipped 2 previous similar messages [ 2754.023582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2759.138870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2759.151266] Lustre: Skipped 2 previous similar messages [ 2761.187049] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2761.405118] Lustre: server umount lustre-MDT0000 complete [ 2764.258138] LustreError: 88022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.278443] LustreError: 88022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2765.194371] Lustre: server umount lustre-MDT0001 complete [ 2768.572863] Lustre: server umount MGS complete [ 2778.743845] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:32:57 (1788471177) [ 2808.748611] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 2817.953106] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 2854.888240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2860.312986] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2870.566133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2870.856547] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2870.942172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2871.064975] Lustre: lustre-MDT0001: new disk, initializing [ 2871.171212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2871.226234] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2871.240629] Lustre: 93651:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2871.257737] LustreError: 93651:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2875.978257] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2887.508241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2887.943948] Lustre: lustre-OST0000: new disk, initializing [ 2887.951219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2887.960678] Lustre: 94419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2888.074875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2894.530191] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2895.970361] Lustre: 94793:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000073da20c2 x1875347835497472/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788471312 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2903.669280] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2910.625541] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.643287] Lustre: MGS: Client e54ed4b0-12d0-412a-9b35-7483e364936a (at 0@lo) reconnecting [ 2910.713627] Lustre: 92918:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2910.742579] LustreError: 92918:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2910.749580] LustreError: 92918:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2910.779768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2910.927978] Lustre: lustre-MDT0000: new disk, initializing [ 2911.156242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2911.200235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2911.229951] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2912.299891] Lustre: 95487:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000cbd1d150 x1875347835504768/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788471328 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2915.971596] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 2917.473987] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2920.433367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2920.440616] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2921.521843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2921.536198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2921.574288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2947.554060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2947.562590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.582453] Lustre: Skipped 1 previous similar message [ 2947.587461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2958.305534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2958.310380] Lustre: Skipped 4 previous similar messages [ 2960.353438] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2960.714253] Lustre: server umount lustre-OST0000 complete [ 2965.473255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2965.478677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.501551] Lustre: Skipped 1 previous similar message [ 2971.039908] Lustre: server umount lustre-MDT0000 complete [ 2973.665838] LustreError: 93623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2973.695801] LustreError: 93623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2981.432878] Lustre: server umount lustre-MDT0001 complete [ 2985.367332] Lustre: server umount MGS complete [ 2997.326422] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:36:35 (1788471395) [ 3017.142875] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 3019.891707] Key type lgssc unregistered [ 3020.174627] LNet: 97988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3020.185375] LNetError: 97988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3020.207195] LNet: Removed LNI 192.168.206.134@tcp [ 3021.144245] Key type .llcrypt unregistered [ 3021.147095] Key type ._llcrypt unregistered [ 3036.173762] Key type ._llcrypt registered [ 3036.179138] Key type .llcrypt registered [ 3036.310557] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 3037.732685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3037.762360] alg: No test for adler32 (adler32-zlib) [ 3038.781291] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 3039.000246] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 3040.704308] Key type lgssc registered [ 3042.026835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3049.807599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3049.828749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.367678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3055.474547] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3060.715390] LustreError: 99184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.734156] LustreError: 99184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3064.538845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.020324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3069.839773] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3077.722218] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3083.548236] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3090.984931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3091.376170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3095.508135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3098.644022] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3107.039404] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3120.107283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.113770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3120.120455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3125.218251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3125.232575] Lustre: Skipped 2 previous similar messages [ 3126.035796] Lustre: server umount lustre-OST0000 complete [ 3130.338267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.358361] Lustre: Skipped 1 previous similar message [ 3130.368294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.374170] Lustre: Skipped 1 previous similar message [ 3135.458523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3135.470723] Lustre: Skipped 1 previous similar message [ 3140.585960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.594700] Lustre: Skipped 1 previous similar message [ 3144.160293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3144.375342] Lustre: server umount lustre-MDT0000 complete [ 3145.701023] LustreError: 101450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3145.715432] LustreError: 101450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3148.443242] LustreError: 99917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471549 with bad export cookie 8695294010084809586 [ 3148.453127] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3148.460380] LustreError: 99917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3148.984518] Lustre: server umount lustre-MDT0001 complete [ 3160.159820] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 3162.945238] Key type lgssc unregistered [ 3163.203159] LNet: 102738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3163.210418] LNetError: 102738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3164.270868] LNet: Removed LNI 192.168.206.134@tcp [ 3165.074106] Key type .llcrypt unregistered [ 3165.075961] Key type ._llcrypt unregistered [ 3180.909765] Key type ._llcrypt registered [ 3180.911248] Key type .llcrypt registered [ 3180.980169] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 3181.820295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3181.922337] alg: No test for adler32 (adler32-zlib) [ 3182.984391] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 3183.227381] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 3184.912176] Key type lgssc registered [ 3186.097127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3191.925118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3191.931834] LustreError: 103818:0:(tgt_mount.c:2652:server_fill_super()) cfs_race id 716 sleeping [ 3191.937112] LustreError: 103819:0:(tgt_mount.c:2652:server_fill_super()) cfs_fail_race id 716 waking [ 3191.942663] LustreError: 103818:0:(tgt_mount.c:2652:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 3191.949171] LustreError: 103818:0:(tgt_mount.c:2664:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3191.956353] LustreError: 103818:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3191.975133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.452972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3200.135793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.443481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3204.981888] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3210.554274] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing lsmod [ 3213.600764] LustreError: 105023:0:(tgt_mount.c:2652:server_fill_super()) cfs_race id 716 sleeping [ 3213.618335] LustreError: 105026:0:(tgt_mount.c:2652:server_fill_super()) cfs_fail_race id 716 waking [ 3213.622689] LustreError: 105023:0:(tgt_mount.c:2652:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3213.634152] LustreError: 105023:0:(tgt_mount.c:2664:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3213.639427] LustreError: 105023:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3213.660875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3214.044338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.425723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.433504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3219.458884] Lustre: Skipped 2 previous similar messages [ 3219.470031] Lustre: Skipped 1 previous similar message [ 3223.507600] Lustre: server umount lustre-MDT0000 complete [ 3224.545396] LustreError: 103856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.570260] LustreError: 103856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3225.589660] LustreError: 103857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.607853] LustreError: 103857:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3227.485128] LustreError: 103832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471628 with bad export cookie 17575907058812563663 [ 3227.494948] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3228.234955] Lustre: server umount lustre-MDT0001 complete [ 3237.985192] Lustre: 105487:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3243.378072] Lustre: server umount lustre-OST0000 complete [ 3252.671131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.034787] LustreError: 106094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3253.109191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3257.228589] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3258.337734] LustreError: 106095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.444429] LustreError: 106095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3265.571564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.031297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3270.546154] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3279.337710] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3285.713399] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3293.512436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3294.015789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3298.169835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3300.392827] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3308.175751] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3320.810403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3320.813665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3320.819511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3320.832318] Lustre: Skipped 1 previous similar message [ 3320.850961] Lustre: Skipped 2 previous similar messages [ 3325.481213] Lustre: server umount lustre-OST0000 complete [ 3331.041734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.055662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3331.060910] Lustre: Skipped 1 previous similar message [ 3336.163160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3336.175280] Lustre: Skipped 1 previous similar message [ 3341.287593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3341.294017] Lustre: Skipped 1 previous similar message [ 3343.328120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3343.500432] Lustre: server umount lustre-MDT0000 complete [ 3346.401601] LustreError: 106094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3346.413988] LustreError: 106094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3346.830650] LustreError: 106076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471747 with bad export cookie 17575907058812564391 [ 3346.840272] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3346.843697] LustreError: 106076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3347.268955] Lustre: server umount lustre-MDT0001 complete [ 3357.702830] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 3360.268503] Key type lgssc unregistered [ 3360.687333] LNet: 109630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3360.705075] LNetError: 109630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3360.722287] LNet: Removed LNI 192.168.206.134@tcp [ 3361.506898] Key type .llcrypt unregistered [ 3361.512389] Key type ._llcrypt unregistered [ 3385.723274] Key type ._llcrypt registered [ 3385.729097] Key type .llcrypt registered [ 3385.814734] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 3386.768532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3386.824501] alg: No test for adler32 (adler32-zlib) [ 3387.867853] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 3388.068590] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 3389.736645] Key type lgssc registered [ 3390.677112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3397.352988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3397.368210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3398.887418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.226620] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3408.355329] LustreError: 111049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.390039] LustreError: 111049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3411.558767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3411.948113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3416.299797] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3424.004550] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3429.252338] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3436.511435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3436.897988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3442.720756] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3444.087978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3450.145408] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3459.206395] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3462.737140] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3466.727152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3466.740337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3466.748928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3466.779880] Lustre: Skipped 1 previous similar message [ 3471.848715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3471.860273] Lustre: Skipped 2 previous similar messages [ 3472.734611] Lustre: server umount lustre-OST0000 complete [ 3476.963438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3476.988351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3476.998018] Lustre: Skipped 1 previous similar message [ 3482.080870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3482.089572] Lustre: Skipped 1 previous similar message [ 3482.516652] Lustre: server umount lustre-MDT0000 complete [ 3486.302969] LustreError: 111029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471886 with bad export cookie 7204442610061384696 [ 3486.313132] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3486.317979] LustreError: 111029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3486.713298] Lustre: server umount lustre-MDT0001 complete [ 3497.512238] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 3500.280720] Key type lgssc unregistered [ 3500.578338] LNet: 114688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3500.591767] LNetError: 114688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3500.605855] LNet: Removed LNI 192.168.206.134@tcp [ 3501.486711] Key type .llcrypt unregistered [ 3501.489371] Key type ._llcrypt unregistered [ 3520.288499] Key type ._llcrypt registered [ 3520.293347] Key type .llcrypt registered [ 3520.480683] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 3521.554813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3521.591348] alg: No test for adler32 (adler32-zlib) [ 3522.766887] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 3523.013251] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 3524.688151] Key type lgssc registered [ 3526.131850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3534.919310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3534.968477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3536.311465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3541.349889] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3545.571292] LustreError: 115907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3545.582302] LustreError: 115907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3550.299628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.597651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3555.462588] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3563.535978] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3569.450341] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3576.909424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3577.344936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.423342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3585.269842] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3594.591967] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3605.988604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3605.992936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3606.012906] Lustre: Skipped 1 previous similar message [ 3606.031781] Lustre: Skipped 1 previous similar message [ 3611.107768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3611.112471] Lustre: Skipped 1 previous similar message [ 3616.230043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3616.244019] Lustre: Skipped 1 previous similar message [ 3617.250453] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3617.434871] Lustre: server umount lustre-OST0000 complete [ 3621.347787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3621.366244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3621.374301] Lustre: Skipped 1 previous similar message [ 3626.467404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3626.479271] Lustre: Skipped 1 previous similar message [ 3627.404769] Lustre: server umount lustre-MDT0000 complete [ 3631.132577] LustreError: 116641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472031 with bad export cookie 7182182964610708981 [ 3631.137752] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3631.143667] LustreError: 116641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3631.461973] Lustre: server umount lustre-MDT0001 complete [ 3642.359946] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 3644.878226] Key type lgssc unregistered [ 3645.170808] LNet: 119439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3645.178862] LNetError: 119439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3645.210782] LNet: Removed LNI 192.168.206.134@tcp [ 3646.020591] Key type .llcrypt unregistered [ 3646.024523] Key type ._llcrypt unregistered [ 3672.716774] Key type ._llcrypt registered [ 3672.718673] Key type .llcrypt registered [ 3672.854682] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 3674.527907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3674.579198] alg: No test for adler32 (adler32-zlib) [ 3675.787143] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 3676.140384] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 3678.024279] Key type lgssc registered [ 3679.691831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3691.001628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3691.019708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.632943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3697.876550] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3702.754540] LustreError: 120854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3702.774968] LustreError: 120854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3707.875606] LustreError: 120855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3708.535166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3709.060487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3714.338550] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3721.607951] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3728.007279] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3735.404156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3735.667181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3737.712794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3743.078785] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3752.781840] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3759.444181] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3763.470880] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3763.477163] Lustre: Skipped 2 previous similar messages [ 3767.106779] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3767.145818] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3768.321521] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3768.327909] Lustre: Skipped 2 previous similar messages [ 3777.209189] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3777.213713] Lustre: Skipped 1 previous similar message [ 3782.479231] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.34@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3782.489144] Lustre: Skipped 2 previous similar messages [ 3787.403493] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.34@tcp 192.168.0.[2,10]@tcp [ 3795.861176] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.34/24@tcp in log lustre-MDT0000 [ 3795.867861] Lustre: Skipped 2 previous similar messages [ 3802.860386] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.34/24@tcp [ 3802.864757] Lustre: Skipped 1 previous similar message [ 3814.882871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3814.891627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3814.907845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3815.906730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3815.913363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3815.921115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3815.953238] Lustre: Skipped 1 previous similar message [ 3821.028336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3821.032050] Lustre: Skipped 1 previous similar message [ 3825.120279] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3825.308844] Lustre: server umount lustre-OST0000 complete [ 3831.269335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.276223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3831.295386] Lustre: Skipped 1 previous similar message [ 3831.314248] Lustre: Skipped 1 previous similar message [ 3835.644189] Lustre: server umount lustre-MDT0000 complete [ 3836.396744] LustreError: 122356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.414033] LustreError: 122356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3839.733568] LustreError: 123891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472240 with bad export cookie 9289771569670288362 [ 3839.740384] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3840.026607] Lustre: server umount lustre-MDT0001 complete [ 3851.909478] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 3854.990446] Key type lgssc unregistered [ 3855.394307] LNet: 125996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3855.398697] LNetError: 125996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3855.457394] LNet: Removed LNI 192.168.206.134@tcp [ 3856.553769] Key type .llcrypt unregistered [ 3856.560833] Key type ._llcrypt unregistered [ 3888.148494] Key type ._llcrypt registered [ 3888.152823] Key type .llcrypt registered [ 3888.284654] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 3889.479989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3889.536592] alg: No test for adler32 (adler32-zlib) [ 3890.601645] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 3890.847593] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 3892.528792] Key type lgssc registered [ 3893.735704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3902.412250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3902.447405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3903.970161] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3903.994893] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.34/24@tcp [ 3904.037828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3908.987842] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3914.213912] LustreError: 127557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3914.228101] LustreError: 127557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3917.968297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.411750] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3918.446110] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.34/24@tcp [ 3918.494829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3922.907305] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3930.653243] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3936.791863] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3943.693922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3943.991370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3949.099470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3949.931360] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 3958.363981] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3971.054382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3971.059978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3971.068695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3973.606566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.615323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3973.622454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3977.252351] Lustre: server umount lustre-OST0000 complete [ 3983.842149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3983.846869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3983.857949] Lustre: Skipped 1 previous similar message [ 3983.869809] Lustre: Skipped 1 previous similar message [ 3987.370350] Lustre: server umount lustre-MDT0000 complete [ 3988.961737] LustreError: 128298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.981955] LustreError: 128298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3990.957114] LustreError: 127538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472391 with bad export cookie 4590801489501320133 [ 3990.964832] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3990.972250] LustreError: 127538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3991.317353] Lustre: server umount lustre-MDT0001 complete [ 4002.476662] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 4005.779319] Key type lgssc unregistered [ 4006.057833] LNet: 131145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4006.068344] LNetError: 131145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4006.097764] LNet: Removed LNI 192.168.206.134@tcp [ 4007.069889] Key type .llcrypt unregistered [ 4007.072115] Key type ._llcrypt unregistered [ 4033.344210] Key type ._llcrypt registered [ 4033.346222] Key type .llcrypt registered [ 4033.435689] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 4050.702507] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 4052.066555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4052.092669] alg: No test for adler32 (adler32-zlib) [ 4053.197961] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 4053.453055] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 4055.248187] Key type lgssc registered [ 4056.220279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4090.843247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4090.874064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4092.158540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4092.202539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4092.301184] Lustre: lustre-MDT0000: new disk, initializing [ 4092.378811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4092.398089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4096.204910] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4106.345715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.431533] Lustre: 134546:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4106.535891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4106.542355] Lustre: Skipped 1 previous similar message [ 4106.655553] Lustre: lustre-MDT0001: new disk, initializing [ 4106.817798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4106.875079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4106.894642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4111.690869] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4121.534250] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4127.322321] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4134.060133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4134.255836] Lustre: lustre-OST0000: new disk, initializing [ 4134.258693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4134.263938] Lustre: 136514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4134.338310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4140.115424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4140.133461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4140.268843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4140.839877] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4150.126663] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4154.509890] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4154.758298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4160.072220] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4160.276561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4166.114528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4166.117320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4166.126023] LustreError: Skipped 1 previous similar message [ 4166.131275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4166.142196] Lustre: Skipped 1 previous similar message [ 4166.156255] Lustre: Skipped 1 previous similar message [ 4171.235017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4171.249536] Lustre: Skipped 1 previous similar message [ 4176.355853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4176.357931] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 4176.364534] Lustre: Skipped 1 previous similar message [ 4176.614663] Lustre: server umount lustre-OST0000 complete [ 4181.477491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.481444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4181.492994] Lustre: Skipped 1 previous similar message [ 4186.597583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4186.602410] Lustre: Skipped 2 previous similar messages [ 4186.927862] Lustre: server umount lustre-MDT0000 complete [ 4190.308367] LustreError: 134538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472590 with bad export cookie 16486372821531426227 [ 4190.312079] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4190.325119] LustreError: 134538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4190.835779] Lustre: server umount lustre-MDT0001 complete [ 4199.136726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4199.588681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4203.580683] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4205.028932] LustreError: 138732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4205.047292] LustreError: 138732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4210.150993] LustreError: 138731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4212.663409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4213.131469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4217.748048] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4225.805509] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4232.382408] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4240.560067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4241.132874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4247.599085] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4255.567314] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4260.224692] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4260.449542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4264.558863] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4264.815679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4274.101123] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4274.361446] Lustre: lustre-OST0001: new disk, initializing [ 4274.365545] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4274.378555] Lustre: 142101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4274.498868] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4281.944408] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4283.796863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4283.824253] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4283.993854] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4290.896473] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4291.144844] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4296.281740] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4296.483807] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4302.538453] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9719041c2000.ost_server_uuid 50 [ 4303.914186] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9719041c2000.ost_server_uuid in IDLE state after 0 sec [ 4314.606512] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4314.613703] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4314.614951] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4314.635191] Lustre: Skipped 1 previous similar message [ 4314.657864] LustreError: Skipped 1 previous similar message [ 4317.372587] Lustre: server umount lustre-OST0001 complete [ 4319.725800] LustreError: 140554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4319.754185] LustreError: 140554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4322.278565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4322.291063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4324.321827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4324.838757] LustreError: 140553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4324.863470] LustreError: 140553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4329.970866] LustreError: 142540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4329.988474] LustreError: 142540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4335.073523] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4335.081039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4335.100276] Lustre: Skipped 7 previous similar messages [ 4335.315430] Lustre: server umount lustre-OST0000 complete [ 4340.197482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4340.228360] Lustre: Skipped 2 previous similar messages [ 4346.052652] Lustre: server umount lustre-MDT0000 complete [ 4350.119746] LustreError: 142100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788472750 with bad export cookie 16486372821531427123 [ 4350.125726] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4350.137505] LustreError: 142100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4350.897407] Lustre: server umount lustre-MDT0001 complete [ 4374.240142] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 4377.097286] Key type lgssc unregistered [ 4377.428581] LNet: 145106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4377.460152] LNetError: 145106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4378.538796] LNet: Removed LNI 192.168.206.134@tcp [ 4379.583535] Key type .llcrypt unregistered [ 4379.586107] Key type ._llcrypt unregistered [ 4418.238446] Key type ._llcrypt registered [ 4418.242493] Key type .llcrypt registered [ 4418.334470] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 4432.180947] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 4433.373886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4433.390397] alg: No test for adler32 (adler32-zlib) [ 4434.419389] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 4434.643157] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 4436.360320] Key type lgssc registered [ 4437.636690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4472.561953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4472.591326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4473.891753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4473.930595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4474.032697] Lustre: lustre-MDT0000: new disk, initializing [ 4474.113555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4474.136662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4478.389523] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4488.951036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4489.044984] Lustre: 149269:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4489.138786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4489.145783] Lustre: Skipped 1 previous similar message [ 4489.262352] Lustre: lustre-MDT0001: new disk, initializing [ 4489.362071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4489.394514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4489.404084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4494.169528] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4503.531848] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4509.635058] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4516.452033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4516.822544] Lustre: lustre-OST0000: new disk, initializing [ 4516.826759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4516.834261] Lustre: 151236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4516.946378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4523.428810] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4527.234817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4527.269372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4527.476765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4533.250444] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4538.102520] Lustre: Failing over lustre-OST0000 [ 4538.199536] Lustre: server umount lustre-OST0000 complete [ 4556.263429] Lustre: 147483:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788472940/real 1788472940] req@00000000ac8aef2a x1875349691467008/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788472956 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4556.316697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4556.842166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4557.432601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4557.474413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4557.488760] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4558.436550] Lustre: 147481:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788472943/real 1788472943] req@00000000492a1528 x1875349691468416/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788472959 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4561.512819] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4563.366529] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4563.552181] Lustre: 147483:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788472948/real 1788472948] req@000000004f7151e9 x1875349691469568/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788472964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4563.602547] Lustre: 147483:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4566.534314] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4566.534706] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4566.944281] Lustre: Failing over lustre-MDT0000 [ 4567.189390] Lustre: server umount lustre-MDT0000 complete [ 4568.992185] Lustre: 147483:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788472953/real 1788472953] req@0000000014ad0832 x1875349691470592/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788472969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4571.623591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4571.633193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4576.761539] LustreError: 149276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4576.777420] LustreError: 149276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4577.261084] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4577.277346] Lustre: 153153:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4581.873568] LustreError: 149275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4581.896763] LustreError: 149275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4582.370640] Lustre: 153153:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4585.324731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4585.705943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4585.743935] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4586.989062] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4589.306826] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4591.110519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4591.116858] Lustre: Skipped 1 previous similar message [ 4591.127886] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4591.159172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4596.705206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4596.710902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4596.718301] Lustre: Skipped 2 previous similar messages [ 4596.721819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4597.224855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4601.321041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4601.326927] Lustre: Skipped 1 previous similar message [ 4602.923980] Lustre: server umount lustre-OST0000 complete [ 4606.443587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4606.451676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4606.461242] Lustre: Skipped 1 previous similar message [ 4606.471875] LustreError: Skipped 1 previous similar message [ 4606.479733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4606.497690] Lustre: Skipped 1 previous similar message [ 4611.555211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4611.563913] Lustre: Skipped 1 previous similar message [ 4612.717776] Lustre: server umount lustre-MDT0000 complete [ 4616.311170] LustreError: 149261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473016 with bad export cookie 3915408892498853084 [ 4616.316352] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4616.322102] LustreError: 149261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4616.766155] Lustre: server umount lustre-MDT0001 complete [ 4626.918807] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 4629.634243] Key type lgssc unregistered [ 4629.881807] LNet: 154906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4629.904368] LNetError: 154906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4629.923407] LNet: Removed LNI 192.168.206.134@tcp [ 4630.754204] Key type .llcrypt unregistered [ 4630.756130] Key type ._llcrypt unregistered [ 4664.164627] Key type ._llcrypt registered [ 4664.167283] Key type .llcrypt registered [ 4664.306943] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 4665.379357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4665.418912] alg: No test for adler32 (adler32-zlib) [ 4666.428810] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 4666.680185] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 4668.376203] Key type lgssc registered [ 4669.478922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4676.266571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4676.282364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.676206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4681.185261] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4687.842540] LustreError: 156422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4687.854959] LustreError: 156422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4689.523071] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4692.958099] LustreError: 156451:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4692.968147] LustreError: 156451:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4692.974183] LustreError: 156451:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 4693.176660] Lustre: server umount lustre-MDT0000 complete [ 4705.743594] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 4714.804441] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 4751.145769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.412817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4751.446970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4751.554653] Lustre: lustre-MDT0000: new disk, initializing [ 4751.633363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4751.655949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4756.960741] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4767.823892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4767.926438] Lustre: 159754:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4768.005562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4768.009558] Lustre: Skipped 1 previous similar message [ 4768.075118] Lustre: lustre-MDT0001: new disk, initializing [ 4768.151372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4768.194647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4768.204710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4773.233913] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4782.371623] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4787.594962] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4794.154354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4794.481540] Lustre: lustre-OST0000: new disk, initializing [ 4794.487067] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4794.493125] Lustre: 161723:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4794.572401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4800.642988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4800.659200] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4800.765916] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 4800.792559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4810.654012] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4814.083836] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4815.985478] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:06:54 (1788473214) [ 5082.593046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5082.598276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5082.619710] Lustre: Skipped 1 previous similar message [ 5082.636249] Lustre: Skipped 2 previous similar messages [ 5087.712652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5087.724140] Lustre: Skipped 2 previous similar messages [ 5090.119695] Lustre: server umount lustre-MDT0000 complete [ 5092.845699] LustreError: 159760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5092.863451] LustreError: 159760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5093.962130] LustreError: 159747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473494 with bad export cookie 5172406363022905762 [ 5093.966974] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5093.977781] LustreError: 159747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5097.954623] LustreError: 159761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5097.954807] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.999462] Lustre: Skipped 1 previous similar message [ 5098.010934] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5100.412353] Lustre: server umount lustre-MDT0001 complete [ 5104.099403] Lustre: 163824:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5109.217600] Lustre: 163826:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5110.470184] Lustre: server umount lustre-OST0000 complete [ 5119.126581] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 5129.598377] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 5167.644307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5168.025444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5168.080607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5168.225732] Lustre: lustre-MDT0000: new disk, initializing [ 5168.313051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.350082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5173.199585] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5184.583379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5184.685218] Lustre: 166394:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5184.818678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5184.825690] Lustre: Skipped 1 previous similar message [ 5184.935248] Lustre: lustre-MDT0001: new disk, initializing [ 5185.047073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5185.092138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5185.101688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5189.518268] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5200.897790] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5207.406471] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5214.540965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5214.778833] Lustre: lustre-OST0000: new disk, initializing [ 5214.793524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5214.812918] Lustre: 168362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5214.968816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5223.251683] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5223.566621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5223.587463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5223.836618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5233.267320] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5238.467402] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5238.665735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5243.564406] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5243.767616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5249.511451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5249.514329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5249.519073] Lustre: Skipped 1 previous similar message [ 5249.523339] Lustre: Skipped 1 previous similar message [ 5249.529440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5254.630076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5254.649035] Lustre: Skipped 1 previous similar message [ 5260.257527] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5260.509455] Lustre: server umount lustre-OST0000 complete [ 5266.913454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5266.938265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.967176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5266.988913] Lustre: Skipped 3 previous similar messages [ 5272.517725] Lustre: server umount lustre-MDT0000 complete [ 5275.116295] LustreError: 167557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5275.144314] LustreError: 167557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5276.897779] LustreError: 166386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473677 with bad export cookie 5172406363023064620 [ 5276.921102] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5276.922106] LustreError: 166386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5277.359701] Lustre: server umount lustre-MDT0001 complete [ 5287.414821] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:14:45 (1788473685) [ 5304.502870] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 5315.649702] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 5354.561148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5354.905324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5354.962110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5355.077967] Lustre: lustre-MDT0000: new disk, initializing [ 5355.197088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5355.220871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5360.032680] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5371.628857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.771790] Lustre: 173124:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5372.308350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5377.374686] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5388.596307] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5395.081066] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5402.953115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5403.294645] Lustre: lustre-OST0000: new disk, initializing [ 5403.298831] Lustre: Skipped 1 previous similar message [ 5403.309738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5403.317449] Lustre: Skipped 2 previous similar messages [ 5403.322573] Lustre: 175103:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5410.542385] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5410.959702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5410.979297] Lustre: Skipped 1 previous similar message [ 5410.982220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5411.176339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5422.369683] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5431.782207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5431.785531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5431.787545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5431.796498] Lustre: Skipped 2 previous similar messages [ 5431.827254] Lustre: Skipped 3 previous similar messages [ 5437.191408] Lustre: server umount lustre-OST0000 complete [ 5442.033925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.052616] Lustre: Skipped 1 previous similar message [ 5456.352204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5456.662767] Lustre: server umount lustre-MDT0000 complete [ 5457.380365] LustreError: 173130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5457.407358] LustreError: 173130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5460.746545] LustreError: 173117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788473861 with bad export cookie 5172406363023065516 [ 5460.753208] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5460.757961] LustreError: 173117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5461.304704] Lustre: server umount lustre-MDT0001 complete [ 5471.635948] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:17:49 (1788473869) [ 5489.025996] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 5504.578992] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 5547.477655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5547.778600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5547.812808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5547.880246] Lustre: lustre-MDT0000: new disk, initializing [ 5547.943627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5547.948394] Lustre: Skipped 2 previous similar messages [ 5547.957735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5552.326844] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5564.002454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5564.145814] Lustre: 179442:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5564.501021] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5572.309824] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5586.563046] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5593.495993] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5600.563601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5600.963366] Lustre: 181420:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5607.768504] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5609.573305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5609.692465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5617.782457] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5630.436151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5630.439928] LustreError: Skipped 1 previous similar message [ 5630.449702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5630.471242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5630.482388] Lustre: Skipped 9 previous similar messages [ 5631.833700] Lustre: server umount lustre-OST0000 complete [ 5642.085979] Lustre: server umount lustre-MDT0000 complete [ 5645.793264] LustreError: 179455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5645.817767] LustreError: 179455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5645.834045] LustreError: 179435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474046 with bad export cookie 5172406363023066615 [ 5645.835312] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5645.845100] LustreError: 179435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5646.177622] Lustre: server umount lustre-MDT0001 complete [ 5658.433177] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 5661.665309] Key type lgssc unregistered [ 5662.148959] LNet: 183277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5662.179194] LNetError: 183277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5662.207410] LNet: Removed LNI 192.168.206.134@tcp [ 5663.567102] Key type .llcrypt unregistered [ 5663.570391] Key type ._llcrypt unregistered [ 5692.230665] Key type ._llcrypt registered [ 5692.232464] Key type .llcrypt registered [ 5692.339293] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 5693.378223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5693.540224] alg: No test for adler32 (adler32-zlib) [ 5694.814857] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 5695.177691] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 5696.977914] Key type lgssc registered [ 5698.643467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5708.220874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5708.269932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5709.700822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5714.594989] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5720.035571] LustreError: 184692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5720.073087] LustreError: 184692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5724.105523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5724.583591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5729.665228] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5738.441671] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5743.717404] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5750.030796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5750.248631] LustreError: 186514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5750.378837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5754.498458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5756.009794] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5762.635726] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5785.570803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.575589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5785.579366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5785.589109] LustreError: Skipped 1 previous similar message [ 5785.592489] Lustre: Skipped 1 previous similar message [ 5787.150730] Lustre: server umount lustre-OST0000 complete [ 5791.213167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5791.221415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5791.229241] Lustre: Skipped 1 previous similar message [ 5791.232939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5795.810310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5795.811390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5795.849412] Lustre: Skipped 1 previous similar message [ 5800.930032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5800.937072] Lustre: Skipped 1 previous similar message [ 5805.024524] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5805.227036] Lustre: server umount lustre-MDT0000 complete [ 5806.056226] LustreError: 185880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5806.083876] LustreError: 185880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5810.021876] LustreError: 184672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474210 with bad export cookie 6426272323079925150 [ 5810.027989] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5810.038231] LustreError: 184672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5810.467641] Lustre: server umount lustre-MDT0001 complete [ 5821.501520] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 5824.369228] Key type lgssc unregistered [ 5824.822768] LNet: 188248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5824.840456] LNetError: 188248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5824.861475] LNet: Removed LNI 192.168.206.134@tcp [ 5825.814803] Key type .llcrypt unregistered [ 5825.822145] Key type ._llcrypt unregistered [ 5851.528604] Key type ._llcrypt registered [ 5851.530816] Key type .llcrypt registered [ 5851.623897] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 5852.564710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5852.594595] alg: No test for adler32 (adler32-zlib) [ 5853.687324] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 5854.017472] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 5855.688212] Key type lgssc registered [ 5856.838276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5864.496233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5864.516251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.929239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.236376] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5876.205360] LustreError: 189660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5876.227406] LustreError: 189660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5879.166884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5879.609814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5884.600353] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5893.617934] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5899.100944] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5905.753836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5906.073602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.263206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5913.751884] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 5922.668307] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5929.954553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5929.963712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5929.974828] Lustre: Skipped 1 previous similar message [ 5929.980907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5930.512537] Lustre: lustre-OST0000: Not available for connect from 192.168.206.34@tcp (stopping) [ 5930.520655] Lustre: Skipped 1 previous similar message [ 5935.081667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5940.194070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5940.207192] Lustre: Skipped 3 previous similar messages [ 5944.289759] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5944.493584] Lustre: server umount lustre-OST0000 complete [ 5953.999289] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff971914c53800.ost_server_uuid 50 [ 5956.035876] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5969.692657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5969.696054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5969.711074] Lustre: Skipped 1 previous similar message [ 5969.711537] Lustre: Skipped 1 previous similar message [ 5976.025559] Lustre: server umount lustre-MDT0000 complete [ 5976.036920] LustreError: 189660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5976.055365] LustreError: 189660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5979.607449] LustreError: 189642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474380 with bad export cookie 8228303050912544361 [ 5979.610040] LustreError: 190391:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 5979.614213] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5979.631861] LustreError: 189642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5979.636983] LustreError: 190391:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 5980.036584] Lustre: server umount lustre-MDT0001 complete [ 5991.757469] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:26:29 (1788474389) [ 5999.009804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.988383] LustreError: 193578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6000.182328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6005.219618] LustreError: 193578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.231376] LustreError: 193578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6005.380937] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6010.341729] LustreError: 193579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.671684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.906230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6019.763684] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6028.327616] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6035.769901] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6044.147384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6044.506504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6045.578511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6051.738636] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6060.481048] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6067.574856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6067.786445] Lustre: lustre-OST0001: new disk, initializing [ 6067.807534] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6067.826100] Lustre: 196419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6067.983233] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6074.821138] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6076.037984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6076.045347] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6076.171563] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6083.951654] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6089.747944] Lustre: lustre-OST0000: Not available for connect from 192.168.206.34@tcp (stopping) [ 6089.756435] Lustre: Skipped 3 previous similar messages [ 6090.725160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6090.731501] LustreError: Skipped 1 previous similar message [ 6090.735707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.752841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6093.091771] Lustre: server umount lustre-OST0000 complete [ 6094.847851] LustreError: 195400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6100.207173] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6100.404075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6105.539242] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6105.845814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6112.253783] LustreError: 195400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6112.289596] LustreError: 195400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6122.466239] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6122.473688] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.498883] Lustre: Skipped 2 previous similar messages [ 6122.499420] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6122.530128] Lustre: Skipped 4 previous similar messages [ 6126.338230] Lustre: server umount lustre-OST0001 complete [ 6132.709567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6132.712885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6132.768510] LustreError: Skipped 1 previous similar message [ 6138.898884] Lustre: server umount lustre-MDT0000 complete [ 6142.441683] LustreError: 193559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474543 with bad export cookie 8228303050912545460 [ 6142.450753] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6142.472802] LustreError: 193559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6142.780111] Lustre: server umount lustre-MDT0001 complete [ 6166.587955] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:29:24 (1788474564) [ 6173.184784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6173.510535] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6173.556390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6173.767929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6178.351588] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6187.436875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6187.540160] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6187.604666] Lustre: 199862:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6192.482230] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6199.937779] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6206.008670] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6212.470350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6212.736247] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6218.355734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6218.922611] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6225.368142] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6231.657904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6231.711208] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6239.547903] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6240.766222] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6248.475460] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6254.049918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6254.059252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.088943] Lustre: Skipped 1 previous similar message [ 6254.106476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6254.112298] Lustre: Skipped 2 previous similar messages [ 6258.056165] Lustre: server umount lustre-OST0000 complete [ 6259.712275] LustreError: 201713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6259.735395] LustreError: 201713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6276.593846] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6276.603880] LustreError: Skipped 1 previous similar message [ 6276.606125] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6276.615725] Lustre: Skipped 1 previous similar message [ 6279.847880] Lustre: server umount lustre-OST0001 complete [ 6290.644697] Lustre: server umount lustre-MDT0000 complete [ 6294.966956] LustreError: 201734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474695 with bad export cookie 8228303050912546923 [ 6294.970510] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6294.980696] LustreError: 201734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6295.455547] Lustre: server umount lustre-MDT0001 complete [ 6316.909815] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:31:55 (1788474715) [ 6330.898801] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 6340.008409] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 6371.556541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6371.746403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6371.779488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6371.874298] Lustre: lustre-MDT0000: new disk, initializing [ 6371.949066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6371.954683] Lustre: Skipped 3 previous similar messages [ 6371.964325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6375.952134] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6385.439855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6385.533291] Lustre: 207959:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6385.591851] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6385.595782] Lustre: Skipped 1 previous similar message [ 6385.646456] Lustre: lustre-MDT0001: new disk, initializing [ 6385.733109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6385.752444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6389.714133] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6398.416452] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6403.615128] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6410.309533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6410.551214] Lustre: lustre-OST0000: new disk, initializing [ 6410.554535] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6410.560664] Lustre: 209925:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6415.880603] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6417.489642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6417.498480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6417.628437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6424.039614] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6428.239923] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6428.429492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6432.842703] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6433.069630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6438.373666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6438.373809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6438.376154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6438.376164] Lustre: Skipped 9 previous similar messages [ 6438.380237] Lustre: Skipped 4 previous similar messages [ 6438.385898] LustreError: Skipped 3 previous similar messages [ 6449.120122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6449.278460] Lustre: server umount lustre-OST0000 complete [ 6459.270438] Lustre: server umount lustre-MDT0000 complete [ 6463.460983] LustreError: 207951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788474864 with bad export cookie 8228303050912548750 [ 6463.467706] LustreError: 207951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6463.470614] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6463.874636] Lustre: server umount lustre-MDT0001 complete [ 6472.974800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6473.594832] LustreError: 212144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6473.618713] LustreError: 212144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6479.033629] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6488.372307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6493.068311] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6501.149432] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6506.875765] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6514.714429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6515.151249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6515.156868] Lustre: Skipped 4 previous similar messages [ 6523.467890] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6533.365270] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6538.640521] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6538.949706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6544.242140] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6544.396187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6548.961244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6548.966718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6548.978550] Lustre: Skipped 3 previous similar messages [ 6552.463852] Lustre: server umount lustre-OST0000 complete [ 6559.797692] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6564.274318] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6592.824853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6600.089643] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6607.967114] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6611.658641] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6613.683264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6613.701665] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6613.702215] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6613.736463] Lustre: Skipped 1 previous similar message [ 6619.112078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6619.123139] Lustre: Skipped 12 previous similar messages [ 6622.691626] Lustre: server umount lustre-OST0000 complete [ 6632.629891] Lustre: server umount lustre-MDT0000 complete [ 6634.471559] LustreError: 212145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6634.480324] LustreError: 212145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6636.261512] LustreError: 212124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475036 with bad export cookie 8228303050912549646 [ 6636.279285] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6636.280155] LustreError: 212124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6636.806840] Lustre: server umount lustre-MDT0001 complete [ 6645.785361] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:37:24 (1788475044) [ 6652.403834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6657.503874] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6667.105585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.120825] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6682.934409] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6688.906711] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6696.008195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6703.506932] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6711.663682] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6716.285525] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6716.515261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6721.182859] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6721.364801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6727.651428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6727.818456] Lustre: lustre-OST0001: new disk, initializing [ 6727.823547] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6727.830711] Lustre: 221479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6733.038480] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6737.144454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6737.155140] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6737.261897] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6742.563536] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6746.679284] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6746.886738] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6750.739107] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6750.914127] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6752.750297] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6752.752910] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6752.754272] LustreError: Skipped 1 previous similar message [ 6752.778447] Lustre: Skipped 5 previous similar messages [ 6758.790660] Lustre: server umount lustre-OST0001 complete [ 6765.350608] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6765.550107] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6769.706753] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6769.952305] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6800.735403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6800.904329] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6800.909968] Lustre: Skipped 5 previous similar messages [ 6806.000605] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6806.027594] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6806.031165] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6806.056287] Lustre: Skipped 1 previous similar message [ 6806.870317] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6814.478645] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6816.118910] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6824.197478] Lustre: server umount lustre-OST0001 complete [ 6855.136154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6855.340256] Lustre: server umount lustre-OST0000 complete [ 6865.273756] Lustre: server umount lustre-MDT0000 complete [ 6869.275580] LustreError: 218091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475269 with bad export cookie 8228303050912550654 [ 6869.286918] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6869.299885] LustreError: 218091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6875.861219] Lustre: server umount lustre-MDT0001 complete [ 6902.445201] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:41:40 (1788475300) [ 6909.768729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.391368] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6910.453673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6917.304691] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6925.783485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6925.890525] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6925.920885] Lustre: 226728:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6932.016612] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6941.019359] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6947.955031] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6956.150292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6956.367292] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6962.293921] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6969.967611] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6982.193067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6982.370147] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6990.114246] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 6998.999216] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7000.897697] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7007.275750] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7007.609530] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7014.379915] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7014.648203] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7021.341103] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9719041c1800.ost_server_uuid 50 [ 7024.109390] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9719041c1800.ost_server_uuid in IDLE state after 0 sec [ 7025.622263] Lustre: Permanently deactivating lustre-OST0001 [ 7025.650388] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7028.736379] Lustre: Permanently reactivating lustre-OST0001 [ 7035.623790] Lustre: server umount lustre-OST0001 complete [ 7036.924545] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7036.926356] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7036.939431] Lustre: Skipped 6 previous similar messages [ 7036.951316] LustreError: 228574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7036.958710] LustreError: Skipped 7 previous similar messages [ 7036.987293] LustreError: 228574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7039.972264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7039.984097] Lustre: Skipped 17 previous similar messages [ 7045.813943] Lustre: server umount lustre-OST0000 complete [ 7056.286657] Lustre: server umount lustre-MDT0000 complete [ 7060.573832] LustreError: 227480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475461 with bad export cookie 8228303050912552012 [ 7060.577444] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7060.584173] LustreError: 227480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7060.961526] Lustre: server umount lustre-MDT0001 complete [ 7087.691552] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:44:46 (1788475486) [ 7095.100838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7095.334416] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7095.370679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7095.378108] Lustre: Skipped 5 previous similar messages [ 7100.364308] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7109.196755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7109.295556] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7109.362571] Lustre: 233424:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7114.004313] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7121.581356] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7128.543939] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7136.174522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7136.418525] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7136.443496] Lustre: Permanently deactivating lustre-OST0000 [ 7136.456474] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7142.549622] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7144.391816] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7152.637228] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7159.542720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7159.608725] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7165.427223] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7166.739400] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7172.270948] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7173.905730] Lustre: Permanently reactivating lustre-OST0000 [ 7173.920675] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7173.927760] Lustre: Skipped 2 previous similar messages [ 7182.487074] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7182.496735] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 0000000055d32710, cur 1788475583 deadline 1788475573 last 1788475543 [ 7182.555196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7194.881195] Lustre: server umount lustre-OST0001 complete [ 7204.081099] Lustre: server umount lustre-OST0000 complete [ 7213.910716] Lustre: server umount lustre-MDT0000 complete [ 7217.475827] LustreError: 235292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475618 with bad export cookie 8228303050912553734 [ 7217.492662] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7217.507595] LustreError: 235292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7217.872946] Lustre: server umount lustre-MDT0001 complete [ 7229.275545] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 7231.977395] Key type lgssc unregistered [ 7232.209902] LNet: 238302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7232.220757] LNetError: 238302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7232.238534] LNet: Removed LNI 192.168.206.134@tcp [ 7233.138297] Key type .llcrypt unregistered [ 7233.146812] Key type ._llcrypt unregistered [ 7258.051532] Key type ._llcrypt registered [ 7258.055856] Key type .llcrypt registered [ 7258.159733] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 7259.650379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7259.703182] alg: No test for adler32 (adler32-zlib) [ 7260.820508] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 7261.140552] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 7262.818820] Key type lgssc registered [ 7264.029883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7272.610541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7272.622519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7274.119273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7278.198113] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7283.694207] LustreError: 239766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7283.720595] LustreError: 239766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7286.477986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7286.586512] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7286.611692] Lustre: 239752:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7286.620175] Lustre: Permanently deactivating lustre-MDT0001 [ 7286.630980] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7286.870032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7291.331791] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7291.935493] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7298.457891] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7303.204542] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7309.099946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7309.402486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7311.466283] LustreError: 241606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.476718] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7315.156380] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7316.981016] LustreError: 241606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7319.013421] LustreError: 241742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7319.035814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7319.046380] LustreError: 241742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7321.782385] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7324.137036] LustreError: 241606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7324.151215] LustreError: 241606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7328.641856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7328.919383] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7334.383536] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7334.392161] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7336.144522] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7344.256556] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7348.352840] Lustre: Permanently reactivating lustre-MDT0001 [ 7348.369649] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7348.376138] Lustre: Skipped 2 previous similar messages [ 7350.289956] Lustre: lustre-MDT0001: Client 1bd0f16d-fba4-444c-9e5c-e637adc98fbb (at 192.168.206.34@tcp) reconnecting [ 7356.578667] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7356.725055] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7356.735576] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7356.748126] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7356.774393] LustreError: 243521:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7356.783994] LustreError: 243521:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7356.800516] LustreError: 243521:0:(obd_config.c:2100:class_config_llog_handler()) MGC192.168.206.134@tcp: cfg command failed: rc = -108 [ 7356.802108] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7356.806475] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7356.806475] [ 7356.808669] LustreError: 239757:0:(mgc_request.c:717:do_requeue()) failed processing log: -108 [ 7358.165387] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7363.844947] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7364.005083] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7368.392187] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7368.604543] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7371.318645] Lustre: Permanently deactivating lustre-MDT0001 [ 7371.345131] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7371.349339] Lustre: Skipped 2 previous similar messages [ 7380.343324] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7392.738783] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7392.742066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7392.769323] Lustre: Skipped 3 previous similar messages [ 7392.780215] Lustre: Skipped 3 previous similar messages [ 7393.084241] Lustre: server umount lustre-MDT0000 complete [ 7397.470642] LustreError: 239745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475798 with bad export cookie 15452642340141001575 [ 7397.480938] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7397.493561] LustreError: 239745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7397.858909] LustreError: 239761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7397.873490] LustreError: 239761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7397.995733] Lustre: server umount lustre-MDT0001 complete [ 7403.804991] Lustre: server umount lustre-OST0000 complete [ 7408.098693] Lustre: 244857:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7412.301874] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:50:11 (1788475811) [ 7413.217391] Lustre: 244860:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7418.337577] Lustre: 244857:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7421.504164] Lustre: server umount lustre-OST0001 complete [ 7427.607783] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 7436.218081] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 7477.877619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7478.104293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7478.120736] Lustre: Skipped 2 previous similar messages [ 7478.158466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7478.254291] Lustre: lustre-MDT0000: new disk, initializing [ 7478.352637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7478.392867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7482.514521] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7492.889334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.976617] Lustre: 248078:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7493.057946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7493.061861] Lustre: Skipped 1 previous similar message [ 7493.150158] Lustre: lustre-MDT0001: new disk, initializing [ 7493.273302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7493.331609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7493.338576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7497.308057] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7506.112626] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7512.117572] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7519.662149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7519.897727] Lustre: lustre-OST0000: new disk, initializing [ 7519.901446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7519.913202] Lustre: 250047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7520.036480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7525.366343] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7527.497372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7527.507848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7527.589452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7533.191098] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7534.994219] LustreError: 249990:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7542.842865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7543.076504] Lustre: lustre-OST0001: new disk, initializing [ 7543.085698] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7543.092164] Lustre: 251161:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7545.040772] LustreError: 249990:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7545.061399] LustreError: 250926:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7549.055471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7549.066428] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7549.177271] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7549.382795] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7555.082737] LustreError: 250926:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7559.020819] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7560.560508] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7564.778031] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7564.790809] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7564.794065] LustreError: Skipped 1 previous similar message [ 7564.803860] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7564.846265] Lustre: Skipped 1 previous similar message [ 7565.859238] Lustre: lustre-OST0001: Not available for connect from 192.168.206.34@tcp (stopping) [ 7565.867630] Lustre: Skipped 1 previous similar message [ 7569.533604] Lustre: server umount lustre-OST0001 complete [ 7569.892368] LustreError: 250032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7569.927534] LustreError: 250032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7589.346079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7589.346708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.352479] LustreError: Skipped 1 previous similar message [ 7589.353271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7589.368882] Lustre: Skipped 1 previous similar message [ 7593.163401] Lustre: server umount lustre-OST0000 complete [ 7600.611568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7600.622237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7600.633381] Lustre: Skipped 1 previous similar message [ 7600.648173] Lustre: Skipped 4 previous similar messages [ 7602.756352] Lustre: server umount lustre-MDT0000 complete [ 7605.736495] LustreError: 249990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7605.763030] LustreError: 249990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7607.042778] LustreError: 248070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476007 with bad export cookie 15452642340141004515 [ 7607.049984] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7607.058229] LustreError: 248070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7613.505471] Lustre: server umount lustre-MDT0001 complete [ 7623.288901] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 7625.744926] Key type lgssc unregistered [ 7626.015348] LNet: 253195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7626.025525] LNetError: 253195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7626.045207] LNet: Removed LNI 192.168.206.134@tcp [ 7626.674987] Key type .llcrypt unregistered [ 7626.677334] Key type ._llcrypt unregistered [ 7663.718890] Key type ._llcrypt registered [ 7663.722651] Key type .llcrypt registered [ 7663.846744] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 7664.904561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7664.929750] alg: No test for adler32 (adler32-zlib) [ 7665.959536] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 7666.217983] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 7667.928881] Key type lgssc registered [ 7668.915569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7676.527351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7676.555169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7677.885238] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7677.931781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7678.126328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7681.927212] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7690.174829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7690.322035] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7690.353169] Lustre: 255398:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7690.589720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7695.119905] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7702.923900] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7709.839533] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7716.992790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7717.331979] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7717.701331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7725.365260] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7727.491387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7734.622906] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7789.036151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7789.052593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7789.055811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7789.081120] LustreError: Skipped 1 previous similar message [ 7793.395664] Lustre: server umount lustre-OST0000 complete [ 7797.844252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7811.539520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7811.937758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7817.577757] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7824.891182] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7829.292405] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7829.302513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7829.318225] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7829.333517] Lustre: Skipped 1 previous similar message [ 7865.313427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7865.321423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7865.323667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7865.338996] Lustre: Skipped 2 previous similar messages [ 7865.346033] Lustre: Skipped 1 previous similar message [ 7869.783748] Lustre: server umount lustre-OST0000 complete [ 7875.042395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7875.048602] LustreError: Skipped 1 previous similar message [ 7875.058413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7875.087453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7875.101362] Lustre: Skipped 1 previous similar message [ 7879.805454] Lustre: server umount lustre-MDT0000 complete [ 7880.674192] LustreError: 257880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7880.703800] LustreError: 257880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7883.631131] LustreError: 255391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476284 with bad export cookie 4167302235743566719 [ 7883.632343] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7883.649362] LustreError: 255391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7884.094977] Lustre: server umount lustre-MDT0001 complete [ 7896.226464] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 7899.187686] Key type lgssc unregistered [ 7899.479855] LNet: 260824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7899.491355] LNetError: 260824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7899.518411] LNet: Removed LNI 192.168.206.134@tcp [ 7900.434504] Key type .llcrypt unregistered [ 7900.436802] Key type ._llcrypt unregistered [ 7925.267519] Key type ._llcrypt registered [ 7925.270355] Key type .llcrypt registered [ 7925.393213] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 7926.223942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7926.324893] alg: No test for adler32 (adler32-zlib) [ 7927.404668] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 7927.625400] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 7929.328904] Key type lgssc registered [ 7930.343880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7937.469120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7937.494633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7938.856069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7942.697418] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7949.281627] LustreError: 262239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7949.307198] LustreError: 262239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7950.833172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7951.276770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7955.840418] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7963.926223] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7969.829358] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7976.739400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7977.048915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7983.347746] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 7984.213611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7991.156173] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8019.940685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8019.953133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8019.971987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8021.475744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8021.478337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8021.496243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8021.516102] Lustre: Skipped 1 previous similar message [ 8024.569131] Lustre: server umount lustre-OST0000 complete [ 8028.129386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8028.137244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.155467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8031.716118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8031.718989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8034.197124] Lustre: server umount lustre-MDT0000 complete [ 8036.834848] LustreError: 262968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8036.860398] LustreError: 262968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8037.882445] LustreError: 262221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476438 with bad export cookie 2337237812298685331 [ 8037.890364] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8037.903752] LustreError: 262221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8038.287964] Lustre: server umount lustre-MDT0001 complete [ 8048.307195] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 8050.740909] Key type lgssc unregistered [ 8051.035369] LNet: 266544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8051.046397] LNetError: 266544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8051.064348] LNet: Removed LNI 192.168.206.134@tcp [ 8051.895870] Key type .llcrypt unregistered [ 8051.898596] Key type ._llcrypt unregistered [ 8072.103509] Key type ._llcrypt registered [ 8072.106877] Key type .llcrypt registered [ 8072.238564] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 8073.417165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8073.518597] alg: No test for adler32 (adler32-zlib) [ 8074.577986] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 8074.839695] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 8076.544444] Key type lgssc registered [ 8077.680391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8085.887897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8085.911221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8087.531428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8092.153216] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8097.764644] LustreError: 267773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8097.795567] LustreError: 267773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8101.554917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8101.975308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8106.051845] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8114.946393] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8122.179295] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8131.108459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8131.635796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8132.663482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8138.105159] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8146.337575] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8158.182677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8158.188960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8158.203880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8160.339875] Lustre: server umount lustre-OST0000 complete [ 8167.394109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8167.396384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8167.409259] Lustre: Skipped 1 previous similar message [ 8170.322507] Lustre: server umount lustre-MDT0000 complete [ 8172.526813] LustreError: 267774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8172.547637] LustreError: 267774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8174.174375] LustreError: 267756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476574 with bad export cookie 11869616611581063800 [ 8174.175609] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8174.182127] LustreError: 267756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8183.951947] Lustre: server umount lustre-MDT0001 complete [ 8194.875295] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 8197.855447] Key type lgssc unregistered [ 8198.158862] LNet: 271451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8198.169408] LNetError: 271451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8198.179778] LNet: Removed LNI 192.168.206.134@tcp [ 8199.155793] Key type .llcrypt unregistered [ 8199.161552] Key type ._llcrypt unregistered [ 8223.967803] Key type ._llcrypt registered [ 8223.970524] Key type .llcrypt registered [ 8224.114925] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 8225.291394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8225.392852] alg: No test for adler32 (adler32-zlib) [ 8226.439055] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 8226.726879] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 8228.440173] Key type lgssc registered [ 8229.508374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8237.219574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8237.250144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8238.900650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8243.936470] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8249.314878] LustreError: 272864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8249.336331] LustreError: 272864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8253.408526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8253.775636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8258.002235] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8264.911596] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8271.146512] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8277.610183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8278.061295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8284.220972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8284.319327] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8292.313442] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8319.970826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8319.972543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8319.979231] LustreError: Skipped 1 previous similar message [ 8319.981056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8320.010038] Lustre: Skipped 1 previous similar message [ 8323.001861] Lustre: server umount lustre-OST0000 complete [ 8330.220476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8330.224530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8330.230088] Lustre: Skipped 1 previous similar message [ 8330.242043] Lustre: Skipped 2 previous similar messages [ 8332.649742] Lustre: server umount lustre-MDT0000 complete [ 8335.344068] LustreError: 272865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8335.368640] LustreError: 272865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8336.355500] LustreError: 272845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476737 with bad export cookie 15064395063504113210 [ 8336.358849] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8336.367740] LustreError: 272845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8336.754072] Lustre: server umount lustre-MDT0001 complete [ 8347.090917] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 8349.354467] Key type lgssc unregistered [ 8349.585749] LNet: 277170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8349.599566] LNetError: 277170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8349.632770] LNet: Removed LNI 192.168.206.134@tcp [ 8350.416410] Key type .llcrypt unregistered [ 8350.419658] Key type ._llcrypt unregistered [ 8366.296116] Key type ._llcrypt registered [ 8366.297809] Key type .llcrypt registered [ 8366.398553] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 8367.293471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8367.333231] alg: No test for adler32 (adler32-zlib) [ 8368.361302] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 8368.622582] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 8370.312151] Key type lgssc registered [ 8371.304799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8378.021813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8378.034669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8379.541560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8383.793261] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8390.115862] LustreError: 278400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8390.137586] LustreError: 278400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8392.164758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8392.495502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8396.541329] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8404.075633] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8409.634593] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8416.019155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8416.313284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8421.520242] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8424.505144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8427.377269] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8438.755255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8438.767806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8438.784909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8439.777863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8439.780771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8439.800872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8443.833980] Lustre: server umount lustre-OST0000 complete [ 8448.993304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8449.000400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8449.034267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8449.042453] Lustre: Skipped 1 previous similar message [ 8453.875373] Lustre: server umount lustre-MDT0000 complete [ 8455.146420] LustreError: 278382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8455.173695] LustreError: 278382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8457.904261] LustreError: 278368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476858 with bad export cookie 6086248768900569738 [ 8457.907610] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8457.921126] LustreError: 278368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8458.280414] Lustre: server umount lustre-MDT0001 complete [ 8469.674770] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 8472.342985] Key type lgssc unregistered [ 8472.646957] LNet: 282099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8472.658615] LNetError: 282099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8472.678345] LNet: Removed LNI 192.168.206.134@tcp [ 8473.496165] Key type .llcrypt unregistered [ 8473.508208] Key type ._llcrypt unregistered [ 8484.359343] Key type ._llcrypt registered [ 8484.363836] Key type .llcrypt registered [ 8484.463246] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8501.266208] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 8514.706360] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 8515.718935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8515.759752] alg: No test for adler32 (adler32-zlib) [ 8516.782515] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 8517.061622] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 8518.795088] Key type lgssc registered [ 8519.867261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8553.356249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8553.403054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8554.640772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8554.676713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8554.788435] Lustre: lustre-MDT0000: new disk, initializing [ 8554.884722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8554.901807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8559.208590] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8568.783648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8568.843129] Lustre: 285663:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8568.950363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8568.959244] Lustre: Skipped 1 previous similar message [ 8569.058763] Lustre: lustre-MDT0001: new disk, initializing [ 8569.208097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8569.252813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8569.260471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8573.426960] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8581.856635] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8587.199431] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8594.301768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8594.628076] Lustre: lustre-OST0000: new disk, initializing [ 8594.631516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8594.646622] Lustre: 287638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8594.739464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8600.680913] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8603.255177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8603.268430] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8603.432121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8609.782445] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8613.909079] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8614.111214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8618.106383] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8618.332925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8624.097805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8624.107596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.107688] LustreError: Skipped 1 previous similar message [ 8624.108529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8624.131815] Lustre: Skipped 1 previous similar message [ 8629.221967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.232814] Lustre: Skipped 1 previous similar message [ 8634.336106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8634.350432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8634.361237] Lustre: Skipped 1 previous similar message [ 8634.541233] Lustre: server umount lustre-OST0000 complete [ 8639.457893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8639.469531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.474374] Lustre: Skipped 1 previous similar message [ 8639.487858] Lustre: Skipped 1 previous similar message [ 8644.579111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8644.588206] Lustre: Skipped 1 previous similar message [ 8652.256115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8652.411203] Lustre: server umount lustre-MDT0000 complete [ 8655.587273] LustreError: 285653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477056 with bad export cookie 2552686470011820905 [ 8655.590470] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8655.594961] LustreError: 285653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8655.864976] Lustre: server umount lustre-MDT0001 complete [ 8663.980483] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:11:02 (1788477062) [ 8670.620036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8671.207448] LustreError: 290052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8671.222613] LustreError: 290052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8671.301693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8675.670736] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8676.328526] LustreError: 290053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8680.419719] LustreError: 290052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8684.306293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8684.538241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8688.724397] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8697.799975] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8704.650921] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8711.599450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8712.066208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8719.495696] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8726.524606] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8739.809086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8739.821398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8739.825821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8739.839271] Lustre: Skipped 1 previous similar message [ 8739.842909] Lustre: Skipped 2 previous similar messages [ 8744.933289] LustreError: 291872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8744.948109] LustreError: 291872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8744.996556] Lustre: server umount lustre-OST0000 complete [ 8750.057085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8750.066588] Lustre: Skipped 1 previous similar message [ 8754.440298] Lustre: server umount lustre-MDT0000 complete [ 8755.172549] LustreError: 290781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8755.190499] LustreError: 290781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8757.946094] LustreError: 290033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477158 with bad export cookie 2552686470011821801 [ 8757.946848] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8757.955723] LustreError: 290033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8758.246745] Lustre: server umount lustre-MDT0001 complete [ 8768.307629] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 8771.102595] Key type lgssc unregistered [ 8771.423702] LNet: 293604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8771.435500] LNetError: 293604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8771.454319] LNet: Removed LNI 192.168.206.134@tcp [ 8772.258506] Key type .llcrypt unregistered [ 8772.260360] Key type ._llcrypt unregistered [ 8796.951262] Key type ._llcrypt registered [ 8796.953405] Key type .llcrypt registered [ 8797.095554] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 8809.474347] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 8810.110867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8810.127034] alg: No test for adler32 (adler32-zlib) [ 8811.140237] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 8811.374178] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 8813.024166] Key type lgssc registered [ 8813.996937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8857.389073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8857.423218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8858.681627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8858.734390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8858.831606] Lustre: lustre-MDT0000: new disk, initializing [ 8858.940967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8858.973932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8864.152672] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8875.732978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8875.800774] Lustre: 297406:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8875.960431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8875.964818] Lustre: Skipped 1 previous similar message [ 8876.047991] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8876.108911] Lustre: lustre-MDT0001: new disk, initializing [ 8876.235295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8876.295342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8876.302549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8881.113571] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8890.850607] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8896.742575] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8901.894672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8902.117743] Lustre: lustre-OST2710: new disk, initializing [ 8902.123224] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8902.128555] Lustre: 299374:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8902.192349] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8907.991216] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8909.888413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8909.894918] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8909.991695] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8916.651122] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8922.768485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8922.891749] Lustre: lustre-OST03e8: new disk, initializing [ 8922.895132] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8922.899900] Lustre: 300438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8923.008805] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8927.700287] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 8928.929318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8928.939065] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8929.020292] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8934.166215] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8942.041913] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8942.403761] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8947.917638] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8948.227630] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8953.787599] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8954.024708] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8958.983500] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8959.246670] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8964.716151] Lustre: server umount lustre-MDT0000 complete [ 8965.095416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8965.111283] Lustre: Skipped 3 previous similar messages [ 8968.622636] LustreError: 297397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477369 with bad export cookie 16705009727917695302 [ 8968.628310] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8968.644930] LustreError: 297397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8968.976595] Lustre: server umount lustre-MDT0001 complete [ 8979.107711] Lustre: 302664:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 8982.253989] Lustre: server umount lustre-OST2710 complete [ 8984.547613] Lustre: 302666:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 8985.376133] Lustre: 295203:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788477370/real 1788477370] req@0000000022a010a6 x1875354281115904/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788477386 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8985.403301] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8985.587891] Lustre: server umount lustre-OST03e8 complete [ 9000.305246] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 9010.448663] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 9047.404535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9047.681674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9047.800534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9047.915561] Lustre: lustre-MDT0000: new disk, initializing [ 9048.086199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9048.101324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9053.952929] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9065.546023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9065.719210] Lustre: 305688:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9065.838975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9065.842900] Lustre: Skipped 1 previous similar message [ 9065.946601] Lustre: lustre-MDT0001: new disk, initializing [ 9066.050081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9066.126355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9066.147083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9070.836230] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9079.394923] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9084.635298] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9090.581992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9090.747656] Lustre: lustre-OST0000: new disk, initializing [ 9090.750845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9090.755584] Lustre: 307655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9090.824833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9096.473097] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9098.786992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9098.797645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9098.913277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9104.215838] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9107.842503] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9107.990888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9111.808328] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9111.951884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9114.082223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9114.087177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9114.095544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9114.600765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9119.647627] Lustre: server umount lustre-OST0000 complete [ 9124.845636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9124.881146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9124.887807] Lustre: Skipped 2 previous similar messages [ 9124.906065] Lustre: Skipped 2 previous similar messages [ 9129.440809] Lustre: server umount lustre-MDT0000 complete [ 9129.966237] LustreError: 308114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9130.002918] LustreError: 308114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9133.375852] LustreError: 305679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477534 with bad export cookie 16705009727917697038 [ 9133.383243] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9133.387254] LustreError: 305679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9133.883535] Lustre: server umount lustre-MDT0001 complete [ 9144.306594] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:19:01 (1788477541) [ 9146.781574] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9148.671629] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:19:07 (1788477547) [ 9152.674865] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 9183.302266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9183.491284] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9183.506719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9183.640224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9188.498507] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9197.211731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9197.293735] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9197.328296] Lustre: 311512:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9202.459448] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9223.857741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9224.043794] Lustre: Denying initial registration attempt from nid 192.168.206.134@tcp, specified as failover [ 9224.052809] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9224.065412] LustreError: 312966:0:(tgt_mount.c:2734:server_fill_super()) Unable to start targets: -99 [ 9224.077978] LustreError: 312966:0:(tgt_mount.c:2192:server_put_super()) no obd lustre-OST0000 [ 9224.082261] LustreError: 312966:0:(tgt_mount.c:130:server_deregister_mount()) lustre-OST0000 not registered [ 9224.156454] Lustre: server umount lustre-OST0000 complete [ 9224.160344] LustreError: 312966:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9230.817532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9230.821747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9230.837946] Lustre: Skipped 1 previous similar message [ 9235.937167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9235.944911] Lustre: Skipped 2 previous similar messages [ 9236.234854] Lustre: server umount lustre-MDT0000 complete [ 9239.789585] LustreError: 311502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477640 with bad export cookie 16705009727917697934 [ 9239.810678] LustreError: 311502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9239.816637] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9240.147387] Lustre: server umount lustre-MDT0001 complete [ 9251.276816] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 9254.145351] Key type lgssc unregistered [ 9254.387212] LNet: 314086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9254.402124] LNetError: 314086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9254.432753] LNet: Removed LNI 192.168.206.134@tcp [ 9255.175511] Key type .llcrypt unregistered [ 9255.178348] Key type ._llcrypt unregistered [ 9266.794986] Key type ._llcrypt registered [ 9266.798652] Key type .llcrypt registered [ 9266.912314] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 9268.007448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9268.114881] alg: No test for adler32 (adler32-zlib) [ 9269.181979] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 9269.392503] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 9271.072158] Key type lgssc registered [ 9272.330717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9319.217515] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 9329.096080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9329.134517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9330.421205] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9330.490045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9330.634644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9335.160628] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9344.644095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9344.784113] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9344.828887] Lustre: 316850:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9345.165906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9349.735171] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9357.687715] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9363.849545] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9383.513435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9383.845155] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9384.131271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9391.928746] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9401.369674] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9406.444101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9406.457951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9406.470940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9406.946831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9406.960510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9406.980111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9409.528514] Lustre: server umount lustre-OST0000 complete [ 9416.676376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9416.690061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9416.717064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9419.445392] Lustre: server umount lustre-MDT0000 complete [ 9423.233280] LustreError: 316841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477823 with bad export cookie 2874638561355647851 [ 9423.250329] LustreError: 316841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9423.251880] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9423.545551] Lustre: server umount lustre-MDT0001 complete [ 9434.147227] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 9437.025214] Key type lgssc unregistered [ 9437.318660] LNet: 320422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9437.329404] LNetError: 320422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9437.352284] LNet: Removed LNI 192.168.206.134@tcp [ 9438.161945] Key type .llcrypt unregistered [ 9438.164859] Key type ._llcrypt unregistered [ 9461.058230] Key type ._llcrypt registered [ 9461.061896] Key type .llcrypt registered [ 9461.147865] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 9461.902496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9461.935963] alg: No test for adler32 (adler32-zlib) [ 9462.960553] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 9463.155714] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 9464.816740] Key type lgssc registered [ 9465.689507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9471.984432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9471.999545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9473.408153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9477.237155] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9483.752290] LustreError: 321836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9483.776273] LustreError: 321836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9486.255435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9486.632581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9491.163514] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9498.806150] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9504.766921] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9513.729843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9514.038923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9520.034813] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9527.299284] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9542.627744] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9542.635037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9542.641294] Lustre: Skipped 1 previous similar message [ 9544.220311] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.34@tcp (stopping) [ 9544.241100] Lustre: Skipped 2 previous similar messages [ 9546.992555] Lustre: server umount lustre-MDT0000 complete [ 9547.746360] LustreError: 321833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9547.776327] LustreError: 321833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9549.300128] LustreError: 321831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9550.421761] LustreError: 321818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477951 with bad export cookie 6367128618225945077 [ 9550.434354] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9550.441591] LustreError: 321818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9552.870208] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9552.871739] LustreError: 323078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9552.899331] Lustre: Skipped 1 previous similar message [ 9552.922841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9555.874127] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9555.881668] Lustre: Skipped 1 previous similar message [ 9556.806772] Lustre: server umount lustre-MDT0001 complete [ 9561.000744] Lustre: 324644:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9563.065505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9566.177083] Lustre: 324644:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9570.855709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9571.391638] LustreError: 323669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9571.410396] LustreError: 323669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9571.467887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9571.541220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9575.606451] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9580.011744] LustreError: 325450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9580.031952] LustreError: 325450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9583.258597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9583.596873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9585.645155] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9587.739411] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9594.064278] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9595.386636] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9601.219802] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9602.754579] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9609.188821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9609.206541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9609.239191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9611.239272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9614.184357] Lustre: server umount lustre-OST0000 complete [ 9619.440023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9619.449176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9619.474805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9619.483136] Lustre: Skipped 2 previous similar messages [ 9624.394057] Lustre: server umount lustre-MDT0000 complete [ 9626.593140] LustreError: 325456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9626.608240] LustreError: 325456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9627.708683] LustreError: 321818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478028 with bad export cookie 6367128618225952987 [ 9627.714681] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9627.717386] LustreError: 321818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9628.147859] Lustre: server umount lustre-MDT0001 complete [ 9637.656201] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 9639.998243] Key type lgssc unregistered [ 9640.255872] LNet: 328125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9640.268135] LNetError: 328125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9640.285830] LNet: Removed LNI 192.168.206.134@tcp [ 9641.030760] Key type .llcrypt unregistered [ 9641.033814] Key type ._llcrypt unregistered [ 9662.754831] Key type ._llcrypt registered [ 9662.756670] Key type .llcrypt registered [ 9662.871873] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 9664.135320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9664.233164] alg: No test for adler32 (adler32-zlib) [ 9665.287823] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 9665.574860] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 9667.256146] Key type lgssc registered [ 9668.453544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9676.169022] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9676.211573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9677.598677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9681.849546] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9688.035647] LustreError: 329539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9688.052811] LustreError: 329539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9690.723953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9690.983304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9694.964779] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9703.468583] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9709.307975] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9716.705959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9716.719533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9716.740326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9717.732098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9717.747885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9717.762465] Lustre: Skipped 1 previous similar message [ 9718.761228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9718.768540] Lustre: Skipped 1 previous similar message [ 9719.280950] Lustre: server umount lustre-MDT0000 complete [ 9723.249612] LustreError: 331154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478123 with bad export cookie 17514055960527310832 [ 9723.262323] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9723.268317] LustreError: 331154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9723.719709] Lustre: server umount lustre-MDT0001 complete [ 9741.871783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9742.400429] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9742.449347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9742.761560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9748.517608] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9758.240942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9758.393776] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9758.433706] Lustre: 332126:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9758.732507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9763.856127] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9772.397731] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9778.630812] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9790.520739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9790.735384] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9790.741626] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9790.747977] LustreError: 334010:0:(tgt_mount.c:2734:server_fill_super()) Unable to start targets: -2 [ 9790.751678] LustreError: 334010:0:(tgt_mount.c:2192:server_put_super()) no obd lustre-OST0000 [ 9790.756029] LustreError: 334010:0:(tgt_mount.c:130:server_deregister_mount()) lustre-OST0000 not registered [ 9790.796933] Lustre: server umount lustre-OST0000 complete [ 9790.800728] LustreError: 334010:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9801.799180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9802.052521] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9802.280626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9809.057919] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9809.645434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9817.426125] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9825.472860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9825.536897] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9825.646836] Lustre: lustre-OST0001: new disk, initializing [ 9825.650964] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9825.656199] Lustre: 335555:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9825.732390] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9831.885882] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [ 9832.097621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9832.123454] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9832.274312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9839.676153] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9847.782210] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9847.805955] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9847.811268] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9847.825098] Lustre: Skipped 1 previous similar message [ 9847.843106] Lustre: Skipped 1 previous similar message [ 9852.905123] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9852.909438] Lustre: Skipped 1 previous similar message [ 9856.992421] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9857.229336] Lustre: server umount lustre-OST0001 complete [ 9858.031050] LustreError: 334478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9858.070407] LustreError: 334478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9863.138136] LustreError: 334477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9863.153981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9863.178929] LustreError: 334477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9863.199663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9863.204829] Lustre: Skipped 1 previous similar message [ 9868.267673] LustreError: 334873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9868.324347] LustreError: 334873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9868.741780] Lustre: server umount lustre-OST0000 complete [ 9873.382245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9873.410496] Lustre: Skipped 1 previous similar message [ 9883.618686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9883.624862] Lustre: Skipped 7 previous similar messages [ 9887.200217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9887.466804] Lustre: server umount lustre-MDT0000 complete [ 9888.742977] LustreError: 333635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9888.760594] LustreError: 333635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9890.852159] LustreError: 332118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478291 with bad export cookie 17514055960527311399 [ 9890.863658] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9890.871438] LustreError: 332118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9891.203221] Lustre: server umount lustre-MDT0001 complete [ 9901.696404] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [ 9904.157854] Key type lgssc unregistered [ 9904.409297] LNet: 337513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9904.418965] LNetError: 337513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9904.444925] LNet: Removed LNI 192.168.206.134@tcp [ 9905.229238] Key type .llcrypt unregistered [ 9905.232784] Key type ._llcrypt unregistered [ 9975.338477] Key type ._llcrypt registered [ 9975.341403] Key type .llcrypt registered [ 9975.428767] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [ 9991.033665] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [ 9992.379903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9992.444950] alg: No test for adler32 (adler32-zlib) [ 9993.483733] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 9993.729926] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [ 9995.448993] Key type lgssc registered [ 9996.451838] Lustre: Echo OBD driver; http://www.lustre.org/ [10034.478972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10034.530927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10035.928348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10035.985055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10036.124304] Lustre: lustre-MDT0000: new disk, initializing [10036.290102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10036.302442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10041.690698] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10052.491857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10052.648763] Lustre: 342446:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10052.759592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10052.764085] Lustre: Skipped 1 previous similar message [10052.907696] Lustre: lustre-MDT0001: new disk, initializing [10053.111662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10053.199300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10053.222901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10058.065668] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10068.296541] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10074.429552] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10081.512175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10081.772257] Lustre: lustre-OST0000: new disk, initializing [10081.776524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10081.782143] Lustre: 344415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10081.861259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10089.445386] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10092.174122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10092.193656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10092.329468] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10101.388323] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10106.980461] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10107.215911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10111.165237] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10111.519138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10118.114233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10118.122570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10118.141639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10118.155398] Lustre: Skipped 1 previous similar message [10119.401176] Lustre: server umount lustre-OST0000 complete [10124.769583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10124.779372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10124.808605] Lustre: Skipped 1 previous similar message [10124.820633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10128.354978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10128.356027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10128.390070] Lustre: Skipped 1 previous similar message [10130.179575] Lustre: server umount lustre-MDT0000 complete [10133.897367] LustreError: 342438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478534 with bad export cookie 8238626043816791723 [10133.908630] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10134.337956] Lustre: server umount lustre-MDT0001 complete [10143.401112] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:35:41 (1788478541) [10151.245583] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:35:49 (1788478549) [10160.132068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10160.626140] LustreError: 347168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10160.645763] LustreError: 347168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10160.716169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10165.055159] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10165.734616] LustreError: 347169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10170.853578] LustreError: 347168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10172.729441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10173.226666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10177.243981] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10184.536231] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10190.651761] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10197.668614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10198.036288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10204.865219] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10212.850607] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10220.013688] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10222.018824] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10223.963775] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10225.853935] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10228.047685] Lustre: Failing over lustre-MDT0000 [10228.340219] Lustre: server umount lustre-MDT0000 complete [10229.730375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10229.739521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10229.748296] LustreError: 347169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10235.369422] LustreError: 347165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10235.392785] LustreError: 347165:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10236.899167] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10236.921179] Lustre: 350063:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10242.022572] Lustre: 350063:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10245.616905] LustreError: 349611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10245.642399] LustreError: 349611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10247.138085] Lustre: 350061:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10247.183585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10247.657565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10247.722577] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10250.729872] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10252.381628] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10252.797119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10252.830485] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10252.858359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10262.963818] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10264.711919] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10269.200487] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10271.769299] Lustre: Failing over lustre-MDT0000 [10272.039546] Lustre: server umount lustre-MDT0000 complete [10273.260171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10273.261974] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10273.303574] Lustre: Skipped 3 previous similar messages [10273.315762] LustreError: 349611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10273.337907] LustreError: 349611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10278.373179] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10278.394880] Lustre: 351260:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10283.491184] Lustre: 351258:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10291.666046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10292.130214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10292.186567] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10296.019832] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10296.820589] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10297.330634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10297.339472] Lustre: Skipped 2 previous similar messages [10297.368988] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10297.399655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10304.720532] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10306.338288] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10312.673299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10312.677491] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10312.678617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10312.700979] Lustre: Skipped 3 previous similar messages [10317.785631] Lustre: server umount lustre-MDT0000 complete [10317.799191] LustreError: 347168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10317.827037] LustreError: 347168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [10321.022718] LustreError: 347148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478721 with bad export cookie 8238626043816794600 [10321.023597] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10321.039474] LustreError: 347148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10321.375160] Lustre: server umount lustre-MDT0001 complete [10331.176105] Lustre: 352696:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10331.189569] Lustre: 352696:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [10335.456430] Lustre: server umount lustre-OST0000 complete [10349.892306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10355.238924] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10364.633265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10365.083610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10365.090241] Lustre: Skipped 1 previous similar message [10370.088257] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10378.305227] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10384.326550] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10390.895886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10395.375248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10397.562594] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10405.066852] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10415.082394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10415.086424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10415.100261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10415.113194] Lustre: Skipped 2 previous similar messages [10421.206146] Lustre: server umount lustre-OST0000 complete [10425.321038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10425.338203] Lustre: Skipped 3 previous similar messages [10430.721842] Lustre: server umount lustre-MDT0000 complete [10434.120748] LustreError: 353506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478834 with bad export cookie 8238626043816795391 [10434.132725] LustreError: 353506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10434.133642] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10434.373762] Lustre: server umount lustre-MDT0001 complete [10443.778862] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [10446.031117] Key type lgssc unregistered [10446.352544] LNet: 357057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10446.361550] LNetError: 357057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10446.387259] LNet: Removed LNI 192.168.206.134@tcp [10447.161623] Key type .llcrypt unregistered [10447.165243] Key type ._llcrypt unregistered [10471.354976] Key type ._llcrypt registered [10471.357220] Key type .llcrypt registered [10471.433184] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [10472.290626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10472.393799] alg: No test for adler32 (adler32-zlib) [10473.358763] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [10473.570071] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [10475.216175] Key type lgssc registered [10476.212746] Lustre: Echo OBD driver; http://www.lustre.org/ [10483.313312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10483.327550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10484.717199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10488.430862] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10494.951275] LustreError: 358569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10494.966059] LustreError: 358569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10496.274495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10496.671641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10500.970305] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10508.110941] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10513.455749] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10519.752376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10520.009250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10526.141031] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10530.310552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10532.847476] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10542.567774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10542.575128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10542.587282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10545.639419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10545.657792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10545.664595] Lustre: Skipped 1 previous similar message [10548.107683] Lustre: server umount lustre-MDT0000 complete [10550.754062] LustreError: 358570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10550.775716] LustreError: 358570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10552.208841] LustreError: 358549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478952 with bad export cookie 11104313903828615540 [10552.216448] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10552.231361] LustreError: 358549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10552.524831] Lustre: server umount lustre-MDT0001 complete [10561.697324] Lustre: 361534:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10565.956873] Lustre: server umount lustre-OST0000 complete [10580.359515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10580.698907] LustreError: 362286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10580.792349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10585.811455] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10586.084305] LustreError: 362287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10591.208657] LustreError: 362286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10594.089253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10594.348478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10598.662242] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10605.915546] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10612.025188] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10618.442229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10618.826366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10624.229356] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10626.065282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10631.245220] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10641.379086] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10641.381238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10641.385867] Lustre: Skipped 3 previous similar messages [10641.395467] Lustre: Skipped 2 previous similar messages [10646.499481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10647.600519] Lustre: server umount lustre-MDT0000 complete [10650.718121] LustreError: 362267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479051 with bad export cookie 11104313903828616702 [10650.720213] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10650.728171] LustreError: 362267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10650.930503] Lustre: server umount lustre-MDT0001 complete [10661.281976] Lustre: 365372:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10664.531216] Lustre: server umount lustre-OST0000 complete [10678.629329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10679.031341] LustreError: 366201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10679.089448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10683.171098] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10691.095317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10694.805922] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10701.917300] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10707.057360] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10714.116526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10714.481875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10714.487784] Lustre: Skipped 1 previous similar message [10722.379965] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10723.771766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10731.336881] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10738.705681] Lustre: server umount lustre-OST0000 complete [10742.755453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10742.768219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10742.811194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10742.821190] Lustre: Skipped 2 previous similar messages [10752.548849] Lustre: server umount lustre-MDT0000 complete [10754.538859] LustreError: 367296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10754.564935] LustreError: 367296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10755.558572] Lustre: 357881:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788479139/real 1788479139] req@00000000137ecec7 x1875356023872768/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788479155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10755.601546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10755.616668] Lustre: Skipped 1 previous similar message [10756.165966] LustreError: 366182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479156 with bad export cookie 11104313903828617766 [10756.174127] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10756.183185] LustreError: 366182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10756.503633] Lustre: server umount lustre-MDT0001 complete [10767.556888] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [10770.043155] Key type lgssc unregistered [10770.356113] LNet: 369730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10770.361287] LNetError: 369730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10770.373833] LNet: Removed LNI 192.168.206.134@tcp [10771.247580] Key type .llcrypt unregistered [10771.250821] Key type ._llcrypt unregistered [10795.821948] Key type ._llcrypt registered [10795.823906] Key type .llcrypt registered [10795.928980] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [10796.880826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10796.890859] alg: No test for adler32 (adler32-zlib) [10797.960247] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [10798.206966] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [10799.912224] Key type lgssc registered [10800.939879] Lustre: Echo OBD driver; http://www.lustre.org/ [10807.822649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10807.832446] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10807.836870] LustreError: 371166:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10807.849205] LustreError: 371166:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10807.853201] LustreError: 371166:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10807.857688] LustreError: 371166:0:(tgt_mount.c:2664:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10807.863891] LustreError: 371166:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10815.349965] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10815.355257] LustreError: 371563:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10815.367370] LustreError: 371563:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10815.376123] LustreError: 371563:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10815.381826] LustreError: 371563:0:(tgt_mount.c:2664:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10815.389476] LustreError: 371563:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10834.103274] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [10836.448849] Key type lgssc unregistered [10836.735835] LNet: 372567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10836.741384] LNetError: 372567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10836.763970] LNet: Removed LNI 192.168.206.134@tcp [10837.489713] Key type .llcrypt unregistered [10837.492296] Key type ._llcrypt unregistered [10856.993754] Key type ._llcrypt registered [10856.995950] Key type .llcrypt registered [10857.094582] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [10872.428664] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [10873.327077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10873.337442] alg: No test for adler32 (adler32-zlib) [10874.608442] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [10874.798613] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [10876.464179] Key type lgssc registered [10877.813933] Lustre: Echo OBD driver; http://www.lustre.org/ [10912.792169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10912.826869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10914.139557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10914.182893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10914.295397] Lustre: lustre-MDT0000: new disk, initializing [10914.426892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10914.462463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10918.577156] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10929.175692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10929.252619] Lustre: 375718:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10929.321822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10929.330329] Lustre: Skipped 1 previous similar message [10929.435600] Lustre: lustre-MDT0001: new disk, initializing [10929.567462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10929.615045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10929.627092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10934.389172] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10944.938449] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10951.439951] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10959.811159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10960.179308] Lustre: lustre-OST0000: new disk, initializing [10960.192102] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10960.215931] Lustre: 377690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10960.352208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10968.035133] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [10969.731990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10969.741705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10969.873774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10976.810854] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10981.563990] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10981.836300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10986.005314] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10986.231968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10990.562582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10990.567365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10990.570280] LustreError: Skipped 1 previous similar message [10990.571928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10990.590491] Lustre: Skipped 1 previous similar message [10995.689529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10995.697770] Lustre: Skipped 2 previous similar messages [11000.809390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11000.818412] Lustre: Skipped 1 previous similar message [11002.336268] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11002.553443] Lustre: server umount lustre-OST0000 complete [11006.433515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11006.437496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11006.445045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11011.042195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11011.047549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11012.294480] Lustre: server umount lustre-MDT0000 complete [11015.683191] LustreError: 379232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479416 with bad export cookie 8106412911654108156 [11015.686239] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11015.691147] LustreError: 379232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11016.080136] Lustre: server umount lustre-MDT0001 complete [11026.181423] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11028.242215] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:50:26 (1788479426) [11035.595317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11036.101764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11040.186041] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11041.259179] LustreError: 380152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11041.285189] LustreError: 380152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11045.352456] LustreError: 380151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11048.280481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11048.621528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11052.912670] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11060.186214] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11066.113327] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11073.458511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11073.818027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11080.303489] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11087.936254] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11094.297789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11094.505505] Lustre: lustre-OST0001: new disk, initializing [11094.512356] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11094.517365] Lustre: 382989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11094.589611] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11100.105279] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11104.930545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11104.941450] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11105.002859] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11108.174545] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11115.010244] Lustre: lustre-OST0001: Not available for connect from 192.168.206.34@tcp (stopping) [11115.020773] Lustre: Skipped 1 previous similar message [11115.499211] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11115.505584] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11115.513222] Lustre: Skipped 1 previous similar message [11126.242032] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11126.341439] Lustre: server umount lustre-OST0001 complete [11130.859409] LustreError: 381974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11130.884980] LustreError: 381974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11135.981956] LustreError: 381974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11135.992673] LustreError: 381974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11141.095228] LustreError: 382366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11141.126501] LustreError: 382366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11145.698862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11145.718931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11145.754297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11145.761070] Lustre: Skipped 8 previous similar messages [11149.239592] Lustre: server umount lustre-OST0000 complete [11156.450795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11156.455266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11156.472920] Lustre: Skipped 2 previous similar messages [11158.902762] Lustre: server umount lustre-MDT0000 complete [11161.569473] LustreError: 380148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.592327] LustreError: 380148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11162.517445] LustreError: 380133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479563 with bad export cookie 8106412911654109052 [11162.523873] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11162.532742] LustreError: 380133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11168.852193] Lustre: server umount lustre-MDT0001 complete [11177.966488] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [11180.104754] Key type lgssc unregistered [11180.389444] LNet: 384977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11180.396230] LNetError: 384977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11180.407101] LNet: Removed LNI 192.168.206.134@tcp [11181.106964] Key type .llcrypt unregistered [11181.108967] Key type ._llcrypt unregistered [11223.136294] Key type ._llcrypt registered [11223.139533] Key type .llcrypt registered [11223.243633] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [11224.163905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11224.249059] alg: No test for adler32 (adler32-zlib) [11225.252863] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [11225.437184] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [11227.120179] Key type lgssc registered [11228.168844] Lustre: Echo OBD driver; http://www.lustre.org/ [11235.035904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11235.053810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11236.230431] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11236.245739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11236.355213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11240.671500] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11249.347581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11249.397209] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11249.414125] Lustre: 387770:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11249.593458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11253.951331] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11260.846419] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11265.868651] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11270.129116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11270.137559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11270.155181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11272.164253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11272.173093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11272.191769] Lustre: Skipped 1 previous similar message [11273.938614] Lustre: server umount lustre-MDT0000 complete [11277.726643] LustreError: 387763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479678 with bad export cookie 12765689780274922851 [11277.736226] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11277.747856] LustreError: 387763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11278.230542] Lustre: server umount lustre-MDT0001 complete [11287.339455] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:54:45 (1788479685) [11294.898315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11295.382478] LustreError: 390284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11295.398032] LustreError: 390284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11295.507167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11300.821735] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11300.845542] LustreError: 390285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11305.960640] LustreError: 390284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11310.048594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11310.332310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11314.691852] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11322.195854] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11327.858762] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11334.351309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11334.515258] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11334.675713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11340.617471] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11347.731571] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11355.584918] Lustre: Permanently deactivating lustre-OST0000 [11355.598828] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11365.066436] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11366.593703] LustreError: 393319:0:(mgs_llog.c:2049:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11366.608419] LustreError: 393319:0:(mgs_llog.c:2104:mgs_replace_nids()) Only MGS is allowed to be started [11366.617697] LustreError: 393319:0:(mgs_handler.c:1257:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11369.842679] LustreError: 393367:0:(mgs_llog.c:2104:mgs_replace_nids()) Only MGS is allowed to be started [11369.848948] LustreError: 393367:0:(mgs_handler.c:1257:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11386.336125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11386.637654] Lustre: server umount lustre-OST0000 complete [11389.037883] LustreError: 393618:0:(mgs_llog.c:2104:mgs_replace_nids()) Only MGS is allowed to be started [11389.048981] LustreError: 393618:0:(mgs_handler.c:1257:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11391.457871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11391.461270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11391.467183] Lustre: Skipped 1 previous similar message [11391.481284] Lustre: Skipped 1 previous similar message [11396.577633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11396.589060] Lustre: Skipped 1 previous similar message [11400.800341] Lustre: server umount lustre-MDT0000 complete [11401.698914] LustreError: 390285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11401.716241] LustreError: 390285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11404.278898] LustreError: 392975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479804 with bad export cookie 12765689780274923432 [11404.280281] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11404.287497] LustreError: 392975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11413.638852] Lustre: server umount lustre-MDT0001 complete [11422.028968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11426.108083] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11428.421366] LustreError: 394776:0:(mgs_handler.c:1257:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11433.808546] Lustre: 395115:0:(mgs_llog.c:1814:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.134@tcp [11435.710534] Lustre: server umount MGS complete [11446.681877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11447.095848] LustreError: 395840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11447.109288] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11447.118508] Lustre: Skipped 1 previous similar message [11447.166675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11450.438981] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11457.505661] LustreError: 395840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11457.521705] LustreError: 395840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11457.644708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11457.897413] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11461.595202] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11467.792751] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11472.329759] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11474.003642] Lustre: Permanently reactivating lustre-OST0000 [11474.013441] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11474.016955] Lustre: Skipped 2 previous similar messages [11504.097143] LustreError: 397706:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11510.984933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11511.263466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11511.266888] Lustre: Skipped 1 previous similar message [11516.789070] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11523.043269] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11528.673417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11528.678298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11528.691863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11528.695363] Lustre: Skipped 1 previous similar message [11533.577569] Lustre: server umount lustre-OST0000 complete [11538.917472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11538.925408] Lustre: Skipped 1 previous similar message [11538.937660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11538.948818] Lustre: Skipped 1 previous similar message [11551.200134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11551.394326] Lustre: server umount lustre-MDT0000 complete [11554.284518] LustreError: 397958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11554.305514] LustreError: 397958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11554.714756] LustreError: 395821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479955 with bad export cookie 12765689780274924538 [11554.724056] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11554.736439] LustreError: 395821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11555.282495] Lustre: server umount lustre-MDT0001 complete [11570.035926] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [11573.197406] Key type lgssc unregistered [11573.517758] LNet: 400347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11573.527177] LNetError: 400347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11573.547300] LNet: Removed LNI 192.168.206.134@tcp [11574.624243] Key type .llcrypt unregistered [11574.629268] Key type ._llcrypt unregistered [11591.535869] Key type ._llcrypt registered [11591.539801] Key type .llcrypt registered [11591.693247] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [11607.207947] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [11608.672420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11608.819200] alg: No test for adler32 (adler32-zlib) [11610.305521] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [11610.711936] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [11612.520164] Key type lgssc registered [11614.312547] Lustre: Echo OBD driver; http://www.lustre.org/ [11652.284342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11652.298145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11653.508299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11653.557984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11653.638844] Lustre: lustre-MDT0000: new disk, initializing [11653.721945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11653.736443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11658.090026] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11668.506451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11668.650481] Lustre: 403503:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11668.721209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11668.727040] Lustre: Skipped 1 previous similar message [11668.853810] Lustre: lustre-MDT0001: new disk, initializing [11668.955649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11668.987062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11669.001633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11673.192327] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11682.112411] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11688.057170] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11694.649983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11694.947145] Lustre: lustre-OST0000: new disk, initializing [11694.950632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11694.958457] Lustre: 405479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11695.076238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11700.572023] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11700.870831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11700.890441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11700.986236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11709.082323] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11713.533187] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11713.738235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11718.255610] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11718.434980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11721.699534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11721.702254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11721.719305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11721.734806] Lustre: Skipped 1 previous similar message [11721.747228] Lustre: Skipped 1 previous similar message [11726.635526] Lustre: server umount lustre-OST0000 complete [11730.403433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11730.412619] LustreError: Skipped 1 previous similar message [11730.416956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11730.432434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11731.937995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11731.938985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11731.963161] Lustre: Skipped 1 previous similar message [11736.500940] Lustre: server umount lustre-MDT0000 complete [11740.408541] LustreError: 403496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480141 with bad export cookie 2471361520290300184 [11740.418921] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11740.423402] LustreError: 403496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11740.833542] Lustre: server umount lustre-MDT0001 complete [11750.369238] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11752.537889] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:02:30 (1788480150) [11760.050648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11760.581789] LustreError: 407941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.599503] LustreError: 407941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11760.689857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11764.997747] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11765.731752] LustreError: 407942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11770.864928] LustreError: 407941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11773.954728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11774.288471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11779.108569] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11788.311674] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11794.000335] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11801.263522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11801.518447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11808.365793] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11816.693783] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11818.412356] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11820.911506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11826.147448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11826.151600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11826.159135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11829.217156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11831.192781] Lustre: server umount lustre-OST0000 complete [11835.875745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11835.881288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11835.897487] Lustre: Skipped 1 previous similar message [11835.907761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11835.911558] Lustre: Skipped 2 previous similar messages [11840.831965] Lustre: server umount lustre-MDT0000 complete [11844.011213] LustreError: 407923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480244 with bad export cookie 2471361520290301080 [11844.021855] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11844.025563] LustreError: 407923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11844.270319] Lustre: server umount lustre-MDT0001 complete [11853.424886] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [11856.038412] Key type lgssc unregistered [11856.328732] LNet: 411662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11856.350992] LNetError: 411662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11856.377596] LNet: Removed LNI 192.168.206.134@tcp [11857.269620] Key type .llcrypt unregistered [11857.272061] Key type ._llcrypt unregistered [11880.199219] Key type ._llcrypt registered [11880.201526] Key type .llcrypt registered [11880.327646] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [11880.543298] Key type .llcrypt unregistered [11880.545585] Key type ._llcrypt unregistered [11897.432717] Key type ._llcrypt registered [11897.435629] Key type .llcrypt registered [11897.510549] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [11898.206650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11898.255552] alg: No test for adler32 (adler32-zlib) [11899.308834] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [11899.521342] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [11901.192684] Key type lgssc registered [11902.308388] Lustre: Echo OBD driver; http://www.lustre.org/ [11909.299767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11909.332700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11910.666649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11914.629499] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11920.865513] LustreError: 413868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11920.899182] LustreError: 413868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11922.480919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11922.810944] LustreError: 413869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11922.833062] LustreError: 413869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11922.842107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11928.049766] LustreError: 413868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11928.080581] LustreError: 413868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11929.069505] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11933.155388] LustreError: 413864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11933.180331] LustreError: 413864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11936.246286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11936.876150] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11941.359095] LustreError: 413863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11941.399617] LustreError: 413863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11944.067929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11944.488709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11948.859985] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [11959.779734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11959.794749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11959.814684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11964.801037] Lustre: server umount lustre-OST0000 complete [11969.433855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11969.435646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11969.453212] Lustre: Skipped 2 previous similar messages [11969.455824] Lustre: Skipped 1 previous similar message [11971.686928] Lustre: server umount lustre-MDT0000 complete [11974.629351] LustreError: 415606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11974.644021] LustreError: 415606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11975.681493] LustreError: 413848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480376 with bad export cookie 15223052628019157516 [11975.697187] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11976.007967] Lustre: server umount lustre-MDT0001 complete [11986.002969] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [11988.798943] Key type lgssc unregistered [11989.070353] LNet: 417016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11989.076983] LNetError: 417016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11989.096215] LNet: Removed LNI 192.168.206.134@tcp [11989.904358] Key type .llcrypt unregistered [11989.911564] Key type ._llcrypt unregistered [12012.166402] Key type ._llcrypt registered [12012.170530] Key type .llcrypt registered [12012.293587] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [12013.061793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12013.111997] alg: No test for adler32 (adler32-zlib) [12014.262155] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [12014.547632] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [12016.208205] Key type lgssc registered [12017.275701] Lustre: Echo OBD driver; http://www.lustre.org/ [12024.547330] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12024.561188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12025.665260] Lustre: 418414:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12041.120577] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12046.240533] LustreError: 418401:0:(mgc_request.c:317:do_config_log_add()) MGC192.168.206.134@tcp: failed processing log, type 4: rc = -110 [12050.917611] Lustre: 418416:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12061.681397] Lustre: 418414:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12066.786418] Lustre: 418416:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12071.906973] Lustre: 418414:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12071.970076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12079.532160] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12082.145648] Lustre: 418414:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12082.175407] Lustre: 418414:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [12090.270210] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12100.079401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12100.667452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12100.784802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12105.701041] LustreError: 418431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12105.736742] LustreError: 418431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12106.179935] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12106.728305] LustreError: 418432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12106.750706] LustreError: 418432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12111.849528] LustreError: 418431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12111.860659] LustreError: 418431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12115.117919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12115.501735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12120.736591] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12127.699793] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12132.495485] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12141.038554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12141.044732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12141.047594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12141.053459] Lustre: Skipped 1 previous similar message [12145.135301] Lustre: server umount lustre-OST0000 complete [12151.266124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12151.272267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12151.273940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12151.290745] Lustre: Skipped 2 previous similar messages [12155.069666] Lustre: server umount lustre-MDT0000 complete [12156.385698] LustreError: 419450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12156.397134] LustreError: 419450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12158.266306] LustreError: 418409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480558 with bad export cookie 1444931814392234386 [12158.271787] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12158.278888] LustreError: 418409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12158.676637] Lustre: server umount lustre-MDT0001 complete [12169.918604] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [12172.462371] Key type lgssc unregistered [12172.790189] LNet: 421976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12172.798036] LNetError: 421976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12172.814389] LNet: Removed LNI 192.168.206.134@tcp [12173.705742] Key type .llcrypt unregistered [12173.708238] Key type ._llcrypt unregistered [12196.935482] Key type ._llcrypt registered [12196.937357] Key type .llcrypt registered [12197.026914] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [12198.047992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12198.064626] alg: No test for adler32 (adler32-zlib) [12199.060812] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [12199.330402] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [12200.968267] Key type lgssc registered [12201.989895] Lustre: Echo OBD driver; http://www.lustre.org/ [12208.978281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12208.993377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12210.465715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12214.343595] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12220.903168] LustreError: 423390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12220.911859] LustreError: 423390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12222.480193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12222.848535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12226.961817] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12233.728824] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12238.669530] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12244.649366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12244.954904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12250.539465] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12256.259606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12257.931474] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12263.906218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12263.920712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12263.937118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12264.962799] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.34@tcp (stopping) [12266.465108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12266.468482] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12267.969237] Lustre: server umount lustre-MDT0000 complete [12270.121590] LustreError: 425232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.34@tcp arrived at 1788480670 with bad export cookie 13780782794664794015 [12271.558570] LustreError: 423370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480672 with bad export cookie 13780782794664793091 [12271.564525] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12271.574079] LustreError: 423370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12271.587827] LustreError: 423390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12271.589277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12271.602679] LustreError: 423390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12271.616383] Lustre: Skipped 1 previous similar message [12271.620992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12271.625117] Lustre: Skipped 2 previous similar messages [12276.705565] LustreError: 423391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12276.705748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12276.724166] LustreError: 423391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12277.798430] Lustre: server umount lustre-OST0000 complete [12281.828713] LustreError: 423390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12281.830646] Lustre: 426171:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12281.846804] LustreError: 423390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12284.762358] Lustre: server umount lustre-MDT0001 complete [12296.274754] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [12299.127973] Key type lgssc unregistered [12299.411495] LNet: 427078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12299.417872] LNetError: 427078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12299.435594] LNet: Removed LNI 192.168.206.134@tcp [12300.493850] Key type .llcrypt unregistered [12300.495645] Key type ._llcrypt unregistered [12323.377870] Key type ._llcrypt registered [12323.380679] Key type .llcrypt registered [12323.552160] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [12324.740797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12324.875722] alg: No test for adler32 (adler32-zlib) [12325.940871] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [12326.298698] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [12327.992218] Key type lgssc registered [12329.215618] Lustre: Echo OBD driver; http://www.lustre.org/ [12339.789842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12339.800035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12341.380750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12347.906874] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12350.433718] LustreError: 428490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12350.458268] LustreError: 428490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12355.560391] LustreError: 428491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12357.528984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12358.106335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12363.074782] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12370.675496] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12377.187284] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12384.219145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12384.551212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12385.599634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12390.569135] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12398.755996] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12404.197467] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12404.212044] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12404.224189] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12405.748827] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.34@tcp (stopping) [12406.242129] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12408.557883] Lustre: server umount lustre-MDT0001 complete [12411.363826] LustreError: 428490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12411.384902] LustreError: 428490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12416.481460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12416.485045] LustreError: 429479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12416.500872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12416.501714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12416.501719] Lustre: Skipped 2 previous similar messages [12420.405812] Lustre: server umount lustre-OST0000 complete [12421.606021] LustreError: 428490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12423.970842] Lustre: server umount lustre-MDT0000 complete [12438.291396] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [12441.569911] Key type lgssc unregistered [12441.790730] LNet: 432173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12441.801963] LNetError: 432173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12441.828742] LNet: Removed LNI 192.168.206.134@tcp [12442.805792] Key type .llcrypt unregistered [12442.811716] Key type ._llcrypt unregistered [12467.941154] Key type ._llcrypt registered [12467.945374] Key type .llcrypt registered [12468.144845] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [12482.859915] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [12484.137338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12484.201213] alg: No test for adler32 (adler32-zlib) [12485.309351] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [12485.531112] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [12487.304778] Key type lgssc registered [12488.428783] Lustre: Echo OBD driver; http://www.lustre.org/ [12529.715532] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [12546.752119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12546.795972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12548.193530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12548.247886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12548.369487] Lustre: lustre-MDT0000: new disk, initializing [12548.468895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12548.494774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12553.614312] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12569.865241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12570.204359] Lustre: lustre-OST0000: new disk, initializing [12570.208541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12570.214739] Lustre: Skipped 1 previous similar message [12570.222509] Lustre: 437205:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12570.299579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12576.355882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12576.366173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12576.417620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12576.853713] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12586.571651] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12600.972140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12601.165617] Lustre: 436187:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12601.252983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12601.307934] Lustre: lustre-MDT0001: new disk, initializing [12601.409974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12601.470531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12601.489963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12606.177353] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12615.328407] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12615.596555] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12620.469559] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12620.686214] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12626.913057] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12626.920357] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12626.930785] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12626.946445] Lustre: Skipped 1 previous similar message [12626.954227] Lustre: Skipped 1 previous similar message [12627.460663] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.34@tcp (stopping) [12632.041910] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12632.058474] Lustre: Skipped 1 previous similar message [12637.162662] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12637.171102] Lustre: Skipped 2 previous similar messages [12639.200163] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12639.520653] Lustre: server umount lustre-MDT0001 complete [12647.398262] LustreError: 436193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12647.412380] LustreError: 436193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12652.523916] LustreError: 436193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12652.543424] LustreError: 436193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12657.640367] LustreError: 437517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12657.661459] LustreError: 437517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12658.145911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12658.153074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12658.178168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12658.181795] Lustre: Skipped 1 previous similar message [12662.761914] LustreError: 436198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12663.888201] Lustre: server umount lustre-OST0000 complete [12673.585617] Lustre: server umount lustre-MDT0000 complete [12687.460048] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [12690.164612] Key type lgssc unregistered [12690.505237] LNet: 440954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12690.509376] LNetError: 440954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12690.519804] LNet: Removed LNI 192.168.206.134@tcp [12691.378683] Key type .llcrypt unregistered [12691.381186] Key type ._llcrypt unregistered [12728.780493] Key type ._llcrypt registered [12728.784649] Key type .llcrypt registered [12728.922258] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [12729.283522] Key type .llcrypt unregistered [12729.285631] Key type ._llcrypt unregistered [12744.208146] Key type ._llcrypt registered [12744.210020] Key type .llcrypt registered [12744.340753] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [12745.682724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12745.799768] alg: No test for adler32 (adler32-zlib) [12746.803498] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [12747.131834] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [12748.896146] Key type lgssc registered [12750.358993] Lustre: Echo OBD driver; http://www.lustre.org/ [12776.114147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12776.177675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12777.468316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12777.508528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12777.553321] Lustre: lustre-MDT0000: new disk, initializing [12777.602802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12777.621381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12781.760213] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12792.433472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12792.527378] Lustre: 444321:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12792.632911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12792.636622] Lustre: Skipped 1 previous similar message [12792.695627] Lustre: lustre-MDT0001: new disk, initializing [12792.834563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12792.877414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12792.890354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12797.489713] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12806.271165] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12811.518248] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12817.925404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12818.152645] Lustre: lustre-OST0000: new disk, initializing [12818.156540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12818.162783] Lustre: 446291:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12818.229602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12824.708904] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12827.773049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12827.786759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12827.928687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12834.126561] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12843.491469] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12843.496495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12843.504825] Lustre: Skipped 2 previous similar messages [12843.525351] Lustre: Skipped 2 previous similar messages [12846.955380] Lustre: server umount lustre-MDT0000 complete [12850.269149] LustreError: 444313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788481250 with bad export cookie 9109123567759659333 [12850.282152] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12850.285604] LustreError: 444313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12850.704878] Lustre: server umount lustre-MDT0001 complete [12860.321290] Lustre: 447452:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12863.878852] Lustre: server umount lustre-OST0000 complete [12878.411868] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:21:16 (1788481276) [12887.345760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12887.866374] LustreError: 448324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12887.895156] LustreError: 448324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12887.958806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12892.063853] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12893.158567] LustreError: 448325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12898.275240] LustreError: 448324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12901.516349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12902.005022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12907.120841] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12914.655670] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12920.434990] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12940.711184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12940.863074] Lustre: Found index 0 for lustre-OST0000, updating log [12941.054306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12946.502576] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [12953.671859] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12958.518884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12967.905506] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12967.912417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12967.930067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12968.993592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12969.006347] Lustre: Skipped 2 previous similar messages [12970.218938] Lustre: server umount lustre-OST0000 complete [12988.897130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12988.903834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12988.917238] Lustre: Skipped 1 previous similar message [12988.921452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12992.573045] Lustre: server umount lustre-MDT0000 complete [12994.529455] LustreError: 448324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12994.547309] LustreError: 448324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12996.055159] LustreError: 448304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788481396 with bad export cookie 9109123567759660754 [12996.061948] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12996.069791] LustreError: 448304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12996.386752] Lustre: server umount lustre-MDT0001 complete [12998.048565] LNet: 1 local NIs in recovery (showing 1): 192.168.206.134@tcp [13005.576996] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:23:23 (1788481403) [13626.111228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13626.684460] LustreError: 454739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13626.705611] LustreError: 454739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13626.792947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13631.633476] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [13631.972498] LustreError: 454740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13637.090538] LustreError: 454739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13640.685390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13641.131812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13645.406686] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [13653.044733] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13658.883808] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13677.933569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13678.156922] Lustre: Found index 0 for lustre-OST0000, updating log [13678.687462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13685.062971] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [13688.224649] LNet: 1 local NIs in recovery (showing 1): 192.168.206.134@tcp [13693.300513] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13695.258104] Lustre: server umount lustre-OST0000 complete [13699.751354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13699.754391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13699.762210] Lustre: Skipped 2 previous similar messages [13699.769685] Lustre: Skipped 3 previous similar messages [13700.773561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13700.781038] Lustre: Skipped 1 previous similar message [13701.803379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13701.816860] Lustre: Skipped 1 previous similar message [13703.844693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13703.859062] Lustre: Skipped 3 previous similar messages [13705.224727] Lustre: server umount lustre-MDT0000 complete [13705.893192] LustreError: 454734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13705.904296] LustreError: 454734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13709.927528] LustreError: 454719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788482110 with bad export cookie 9109123567759661860 [13709.930945] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13709.945439] LustreError: 454719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13710.471400] Lustre: server umount lustre-MDT0001 complete [13742.779364] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [13752.391467] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [13790.770383] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:36:29 (1788482189) [13810.210247] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [13813.372590] Key type lgssc unregistered [13813.649498] LNet: 461352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13813.665187] LNetError: 461352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13940.648295] LNet: Removed LNI 192.168.206.134@tcp [13941.568470] Key type .llcrypt unregistered [13941.571425] Key type ._llcrypt unregistered [13960.570795] Key type ._llcrypt registered [13960.572950] Key type .llcrypt registered [13960.802207] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [13962.209811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13962.239572] alg: No test for adler32 (adler32-zlib) [13963.514951] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [13963.859458] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [13965.680151] Key type lgssc registered [13967.258494] Lustre: Echo OBD driver; http://www.lustre.org/ [13973.090758] LNet: Added LNI 192.168.206.134@tcp100 [8/256/0/180] [13974.008239] LNet: Added LNI 192.168.206.134@tcp101 [8/256/0/180] [13974.896629] LNet: Added LNI 192.168.206.134@tcp102 [8/256/0/180] [13975.856826] LNet: Added LNI 192.168.206.134@tcp103 [8/256/0/180] [13977.068799] LNet: Added LNI 192.168.206.134@tcp104 [8/256/0/180] [13977.968763] LNet: Added LNI 192.168.206.134@tcp105 [8/256/0/180] [13978.937479] LNet: Added LNI 192.168.206.134@tcp106 [8/256/0/180] [13979.780543] LNet: Added LNI 192.168.206.134@tcp107 [8/256/0/180] [13980.801524] LNet: Added LNI 192.168.206.134@tcp108 [8/256/0/180] [13981.692785] LNet: Added LNI 192.168.206.134@tcp109 [8/256/0/180] [13982.500610] LNet: Added LNI 192.168.206.134@tcp110 [8/256/0/180] [13983.320641] LNet: Added LNI 192.168.206.134@tcp111 [8/256/0/180] [13984.307957] LNet: Added LNI 192.168.206.134@tcp112 [8/256/0/180] [13985.128562] LNet: Added LNI 192.168.206.134@tcp113 [8/256/0/180] [13985.989434] LNet: Added LNI 192.168.206.134@tcp114 [8/256/0/180] [13986.942385] LNet: Added LNI 192.168.206.134@tcp115 [8/256/0/180] [13987.753103] LNet: Added LNI 192.168.206.134@tcp116 [8/256/0/180] [13988.699657] LNet: Added LNI 192.168.206.134@tcp117 [8/256/0/180] [13989.527342] LNet: Added LNI 192.168.206.134@tcp118 [8/256/0/180] [13990.410622] LNet: Added LNI 192.168.206.134@tcp119 [8/256/0/180] [13991.420334] LNet: Added LNI 192.168.206.134@tcp120 [8/256/0/180] [13992.322379] LNet: Added LNI 192.168.206.134@tcp121 [8/256/0/180] [13993.336059] LNet: Added LNI 192.168.206.134@tcp122 [8/256/0/180] [13994.296871] LNet: Added LNI 192.168.206.134@tcp123 [8/256/0/180] [13995.159724] LNet: Added LNI 192.168.206.134@tcp124 [8/256/0/180] [13996.139677] LNet: Added LNI 192.168.206.134@tcp125 [8/256/0/180] [13996.996196] LNet: Added LNI 192.168.206.134@tcp126 [8/256/0/180] [13997.996955] LNet: Added LNI 192.168.206.134@tcp127 [8/256/0/180] [13998.884649] LNet: Added LNI 192.168.206.134@tcp128 [8/256/0/180] [13999.672899] LNet: Added LNI 192.168.206.134@tcp129 [8/256/0/180] [14000.418292] LNet: Added LNI 192.168.206.134@tcp130 [8/256/0/180] [14001.094215] LNet: Added LNI 192.168.206.134@tcp131 [8/256/0/180] [14001.884801] LNet: Added LNI 192.168.206.134@tcp132 [8/256/0/180] [14002.550526] LNet: Added LNI 192.168.206.134@tcp133 [8/256/0/180] [14003.209981] LNet: Added LNI 192.168.206.134@tcp134 [8/256/0/180] [14004.071789] LNet: Added LNI 192.168.206.134@tcp135 [8/256/0/180] [14009.742133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14009.778115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14010.931930] Lustre: 464381:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.134@tcp: can't add NID 192.168.206.134@tcp130: rc = -75 [14011.156154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14011.188573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14011.285320] Lustre: lustre-MDT0000: new disk, initializing [14011.411890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14011.451849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14012.065102] LNetError: 464409:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.134 [14012.074761] LNetError: Protocol error connecting to 192.168.206.134@tcp114 at host 192.168.206.134:988: is it running a compatible version of Lustre? [14016.381600] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [14027.965940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14028.147689] Lustre: 464399:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14028.285828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14028.295114] Lustre: Skipped 1 previous similar message [14028.394388] Lustre: lustre-MDT0001: new disk, initializing [14028.433387] Lustre: 465201:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.134@tcp: can't add NID 192.168.206.134@tcp130: rc = -75 [14028.442568] Lustre: 465201:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14028.577406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14028.653602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14028.669686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14033.440637] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [14043.020931] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14046.838627] LNetError: 464404:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.134 [14046.854653] LNetError: Protocol error connecting to 192.168.206.134@tcp114 at host 192.168.206.134:988: is it running a compatible version of Lustre? [14048.281462] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14055.634325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14056.083798] Lustre: lustre-OST0000: new disk, initializing [14056.088683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14056.099437] Lustre: 466374:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14056.194897] Lustre: 466379:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.134@tcp: can't add NID 192.168.206.134@tcp130: rc = -75 [14056.209465] Lustre: 466379:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14056.274792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14058.089442] LNetError: 464415:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.134 [14058.109687] LNetError: Protocol error connecting to 192.168.206.134@tcp114 at host 192.168.206.134:988: is it running a compatible version of Lustre? [14061.705439] Lustre: 466686:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.134@tcp: can't add NID 192.168.206.134@tcp130: rc = -75 [14061.713587] Lustre: 466686:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14061.769231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14061.774676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14061.904204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14062.193544] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [14071.393243] LNetError: 461840:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.134 [14071.403590] LNetError: Protocol error connecting to 192.168.206.134@tcp114 at host 192.168.206.134:988: is it running a compatible version of Lustre? [14071.740346] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14082.534886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14082.540206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14082.544095] LustreError: Skipped 1 previous similar message [14082.546088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14082.555311] Lustre: Skipped 1 previous similar message [14087.659749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14087.662379] Lustre: Skipped 2 previous similar messages [14092.778683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14092.792419] Lustre: Skipped 1 previous similar message [14094.304121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14094.433850] Lustre: server umount lustre-OST0000 complete [14097.891058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14097.896831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14097.898971] Lustre: Skipped 1 previous similar message [14098.021131] LNetError: 461841:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.134 [14098.041551] LNetError: Protocol error connecting to 192.168.206.134@tcp114 at host 192.168.206.134:988: is it running a compatible version of Lustre? [14103.010433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14103.015095] Lustre: Skipped 1 previous similar message [14103.610683] Lustre: server umount lustre-MDT0000 complete [14107.353378] LustreError: 464390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788482508 with bad export cookie 14647190946022592368 [14107.370124] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14107.375464] LustreError: 464390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [14107.841060] Lustre: server umount lustre-MDT0001 complete [14117.124791] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [14119.360912] Key type lgssc unregistered [14119.699360] LNet: 468286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14119.715419] LNetError: 468286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14119.722166] LNet: Removed LNI 192.168.206.134@tcp [14120.736282] LNet: Removed LNI 192.168.206.134@tcp100 [14120.753579] LNetError: 468286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14120.761287] LNetError: 468286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14120.771098] LNet: Removed LNI 192.168.206.134@tcp101 [14121.824172] LNet: Removed LNI 192.168.206.134@tcp102 [14121.831936] LNetError: 468286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14121.841977] LNetError: 468286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14121.849764] LNet: Removed LNI 192.168.206.134@tcp103 [14122.912185] LNet: Removed LNI 192.168.206.134@tcp104 [14123.936932] LNet: Removed LNI 192.168.206.134@tcp105 [14123.939177] LNetError: 468286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14123.952503] LNetError: 468286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14123.967619] LNet: Removed LNI 192.168.206.134@tcp106 [14125.027532] LNet: Removed LNI 192.168.206.134@tcp107 [14126.048319] LNet: Removed LNI 192.168.206.134@tcp108 [14127.072302] LNet: Removed LNI 192.168.206.134@tcp109 [14128.096281] LNet: Removed LNI 192.168.206.134@tcp110 [14128.105885] LNetError: 468286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14128.112741] LNetError: 468286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14128.123372] LNet: Removed LNI 192.168.206.134@tcp111 [14129.184159] LNet: Removed LNI 192.168.206.134@tcp112 [14130.208165] LNet: Removed LNI 192.168.206.134@tcp113 [14131.233322] LNet: Removed LNI 192.168.206.134@tcp114 [14132.256156] LNet: Removed LNI 192.168.206.134@tcp115 [14133.280236] LNet: Removed LNI 192.168.206.134@tcp116 [14133.283364] LNet: Removed LNI 192.168.206.134@tcp117 [14133.294733] LNet: Removed LNI 192.168.206.134@tcp118 [14133.308183] LNet: Removed LNI 192.168.206.134@tcp119 [14133.314134] LNet: Removed LNI 192.168.206.134@tcp120 [14133.320452] LNet: Removed LNI 192.168.206.134@tcp121 [14133.323397] LNet: Removed LNI 192.168.206.134@tcp122 [14133.325593] LNet: Removed LNI 192.168.206.134@tcp123 [14133.328237] LNet: Removed LNI 192.168.206.134@tcp124 [14133.331545] LNet: Removed LNI 192.168.206.134@tcp125 [14133.335365] LNet: Removed LNI 192.168.206.134@tcp126 [14133.338496] LNet: Removed LNI 192.168.206.134@tcp127 [14133.343626] LNet: Removed LNI 192.168.206.134@tcp128 [14133.349427] LNet: Removed LNI 192.168.206.134@tcp129 [14133.352634] LNet: Removed LNI 192.168.206.134@tcp130 [14133.355801] LNet: Removed LNI 192.168.206.134@tcp131 [14133.358812] LNet: Removed LNI 192.168.206.134@tcp132 [14133.361634] LNet: Removed LNI 192.168.206.134@tcp133 [14133.365611] LNet: Removed LNI 192.168.206.134@tcp134 [14133.377949] LNet: Removed LNI 192.168.206.134@tcp135 [14133.999664] Key type .llcrypt unregistered [14134.003095] Key type ._llcrypt unregistered [14155.520973] Key type ._llcrypt registered [14155.525817] Key type .llcrypt registered [14155.636740] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [14170.059969] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [14171.099019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14171.147235] alg: No test for adler32 (adler32-zlib) [14172.348862] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [14172.586337] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [14174.256391] Key type lgssc registered [14175.432682] Lustre: Echo OBD driver; http://www.lustre.org/ [14214.121446] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:43:32 (1788482612) [14233.352540] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [14236.501430] Key type lgssc unregistered [14236.782077] LNet: 472368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14236.792916] LNetError: 472368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14236.818381] LNet: Removed LNI 192.168.206.134@tcp [14237.577140] Key type .llcrypt unregistered [14237.579257] Key type ._llcrypt unregistered [14256.810645] Key type ._llcrypt registered [14256.813592] Key type .llcrypt registered [14256.920373] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [14258.043346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14258.080920] alg: No test for adler32 (adler32-zlib) [14259.176945] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [14259.478877] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [14261.136436] Key type lgssc registered [14262.562671] Lustre: Echo OBD driver; http://www.lustre.org/ [14272.333735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14272.386196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14273.689913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14273.736864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14273.794890] Lustre: lustre-MDT0000: new disk, initializing [14273.858537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14273.877811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14278.677386] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [14290.168039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14290.285873] Lustre: 473596:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14290.376562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14290.381479] Lustre: Skipped 1 previous similar message [14290.454885] Lustre: lustre-MDT0001: new disk, initializing [14290.587123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14290.612607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14290.622264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14295.239942] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [14305.025604] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14311.001953] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14319.786201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14320.306549] Lustre: lustre-OST0000: new disk, initializing [14320.312887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14320.331854] Lustre: 475567:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14320.515598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14327.212105] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [14327.509995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14327.531392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14327.772260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14337.534926] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14342.883824] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing lnet_if_list [14349.803308] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14349.836534] LNet: Added LNI 192.168.206.134@tcp10 [8/256/0/180] [14356.043657] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14356.080225] LNet: Added LNI 192.168.206.134@tcp11 [8/256/0/180] [14361.030264] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14361.057480] LNet: Added LNI 192.168.206.134@tcp12 [8/256/0/180] [14365.793522] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14365.830761] LNet: Added LNI 192.168.206.134@tcp13 [8/256/0/180] [14370.846893] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14370.877580] LNet: Added LNI 192.168.206.134@tcp14 [8/256/0/180] [14375.897055] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14375.926538] LNet: Added LNI 192.168.206.134@tcp15 [8/256/0/180] [14380.886989] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14380.910194] LNet: Added LNI 192.168.206.134@tcp16 [8/256/0/180] [14385.556349] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14385.581171] LNet: Added LNI 192.168.206.134@tcp17 [8/256/0/180] [14390.475827] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14390.495563] LNet: Added LNI 192.168.206.134@tcp18 [8/256/0/180] [14396.389598] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14396.452529] LNet: Added LNI 192.168.206.134@tcp19 [8/256/0/180] [14401.755624] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14401.783280] LNet: Added LNI 192.168.206.134@tcp20 [8/256/0/180] [14407.333978] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14407.359779] LNet: Added LNI 192.168.206.134@tcp21 [8/256/0/180] [14413.855599] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14413.898090] LNet: Added LNI 192.168.206.134@tcp22 [8/256/0/180] [14420.470251] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14420.513207] LNet: Added LNI 192.168.206.134@tcp23 [8/256/0/180] [14427.215820] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14427.284532] LNet: Added LNI 192.168.206.134@tcp24 [8/256/0/180] [14434.352936] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14434.390679] LNet: Added LNI 192.168.206.134@tcp25 [8/256/0/180] [14443.223913] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14443.243991] LNet: Added LNI 192.168.206.134@tcp26 [8/256/0/180] [14451.090209] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14451.120777] LNet: Added LNI 192.168.206.134@tcp27 [8/256/0/180] [14459.353056] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14459.387875] LNet: Added LNI 192.168.206.134@tcp28 [8/256/0/180] [14466.504223] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14466.565189] LNet: Added LNI 192.168.206.134@tcp29 [8/256/0/180] [14473.949942] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14473.978380] LNet: Added LNI 192.168.206.134@tcp30 [8/256/0/180] [14481.361963] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14481.400266] LNet: Added LNI 192.168.206.134@tcp31 [8/256/0/180] [14488.348717] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14488.416969] LNet: Added LNI 192.168.206.134@tcp32 [8/256/0/180] [14496.173675] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14496.194787] LNet: Added LNI 192.168.206.134@tcp33 [8/256/0/180] [14503.070551] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14503.111521] LNet: Added LNI 192.168.206.134@tcp34 [8/256/0/180] [14510.511945] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14510.533656] LNet: Added LNI 192.168.206.134@tcp35 [8/256/0/180] [14517.146458] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14517.170334] LNet: Added LNI 192.168.206.134@tcp36 [8/256/0/180] [14524.992543] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14525.018192] LNet: Added LNI 192.168.206.134@tcp37 [8/256/0/180] [14533.143869] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14533.166975] LNet: Added LNI 192.168.206.134@tcp38 [8/256/0/180] [14541.912683] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14541.971353] LNet: Added LNI 192.168.206.134@tcp39 [8/256/0/180] [14550.000637] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14550.049981] LNet: Added LNI 192.168.206.134@tcp40 [8/256/0/180] [14558.140420] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14558.173383] LNet: Added LNI 192.168.206.134@tcp41 [8/256/0/180] [14566.721772] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14566.763285] LNet: Added LNI 192.168.206.134@tcp42 [8/256/0/180] [14574.675756] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14574.702915] LNet: Added LNI 192.168.206.134@tcp43 [8/256/0/180] [14582.920279] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14582.967259] LNet: Added LNI 192.168.206.134@tcp44 [8/256/0/180] [14591.353773] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14591.382457] LNet: Added LNI 192.168.206.134@tcp45 [8/256/0/180] [14602.221692] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14602.252789] LNet: Added LNI 192.168.206.134@tcp46 [8/256/0/180] [14612.194608] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14612.228379] LNet: Added LNI 192.168.206.134@tcp47 [8/256/0/180] [14621.785932] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14621.821841] LNet: Added LNI 192.168.206.134@tcp48 [8/256/0/180] [14630.815430] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14630.868361] LNet: Added LNI 192.168.206.134@tcp49 [8/256/0/180] [14639.154352] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14639.186921] LNet: Added LNI 192.168.206.134@tcp50 [8/256/0/180] [14659.042972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14659.053924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14659.073822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14660.582640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14660.593478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14660.606143] Lustre: Skipped 1 previous similar message [14660.626894] Lustre: Skipped 2 previous similar messages [14664.916358] Lustre: server umount lustre-MDT0000 complete [14669.993309] LustreError: 473587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788483070 with bad export cookie 18430340749369972665 [14670.004045] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14670.021036] LustreError: 473587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14670.458650] Lustre: server umount lustre-MDT0001 complete [14679.985396] Lustre: 498999:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [14685.163328] Lustre: 499001:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [14687.200254] Lustre: 472867:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788483071/real 1788483071] req@00000000dde9a302 x1875359993877760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788483087 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14687.237207] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14687.255466] Lustre: 498999:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [14690.981279] Lustre: 472868:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788483075/real 1788483075] req@00000000061afc53 x1875359993877888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788483091 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14691.362690] Lustre: 499000:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [14692.327468] Lustre: 472866:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788483076/real 1788483076] req@00000000b8651801 x1875359993878016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788483092 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14692.614495] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [14695.395346] Lustre: 472865:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788483080/real 1788483080] req@000000002f77a078 x1875359993878272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788483096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14696.421700] Lustre: 499000:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [14696.452523] Lustre: 499000:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [14700.514215] Lustre: 472867:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788483085/real 1788483085] req@00000000caec16b7 x1875359993878528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788483101 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14704.107396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14704.495485] LustreError: 475560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14704.514490] LustreError: 475560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14704.536300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14709.735463] LustreError: 475561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14709.824945] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [14712.815434] LustreError: 499816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14712.845161] LustreError: 499816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14716.908218] LustreError: 499816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14716.921345] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14716.932146] LustreError: 499816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14719.713669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14720.278837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14724.912363] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [14725.611739] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14733.361462] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14740.624850] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14766.052620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14766.058001] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14766.074121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14768.774853] Lustre: server umount lustre-OST0000 complete [14776.805716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14776.807647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14776.827138] LustreError: Skipped 1 previous similar message [14776.858353] Lustre: Skipped 1 previous similar message [14776.865109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14776.899167] Lustre: Skipped 2 previous similar messages [14780.120439] Lustre: server umount lustre-MDT0000 complete [14781.923071] LustreError: 499820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14781.942685] LustreError: 499820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14784.044778] LustreError: 473588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788483184 with bad export cookie 18430340749369975416 [14784.047078] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14784.056823] LustreError: 473588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14784.415118] Lustre: server umount lustre-MDT0001 complete [14797.913990] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [14800.665602] Key type lgssc unregistered [14801.083658] LNet: 502617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14801.100869] LNetError: 502617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14802.146142] LNet: Removed LNI 192.168.206.134@tcp [14802.150398] LNetError: 502617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14802.170408] LNet: Removed LNI 192.168.206.134@tcp10 [14802.178326] LNet: Removed LNI 192.168.206.134@tcp11 [14802.187443] LNet: Removed LNI 192.168.206.134@tcp12 [14802.198108] LNet: Removed LNI 192.168.206.134@tcp13 [14802.207442] LNet: Removed LNI 192.168.206.134@tcp14 [14802.210786] LNet: Removed LNI 192.168.206.134@tcp15 [14802.214148] LNet: Removed LNI 192.168.206.134@tcp16 [14802.232117] LNet: Removed LNI 192.168.206.134@tcp17 [14802.239953] LNet: Removed LNI 192.168.206.134@tcp18 [14802.243652] LNet: Removed LNI 192.168.206.134@tcp19 [14802.249604] LNet: Removed LNI 192.168.206.134@tcp20 [14802.252120] LNet: Removed LNI 192.168.206.134@tcp21 [14802.258082] LNet: Removed LNI 192.168.206.134@tcp22 [14802.262785] LNet: Removed LNI 192.168.206.134@tcp23 [14802.267091] LNet: Removed LNI 192.168.206.134@tcp24 [14802.270852] LNet: Removed LNI 192.168.206.134@tcp25 [14802.276637] LNet: Removed LNI 192.168.206.134@tcp26 [14802.280140] LNet: Removed LNI 192.168.206.134@tcp27 [14802.283326] LNet: Removed LNI 192.168.206.134@tcp28 [14802.286524] LNet: Removed LNI 192.168.206.134@tcp29 [14802.289311] LNet: Removed LNI 192.168.206.134@tcp30 [14802.292590] LNet: Removed LNI 192.168.206.134@tcp31 [14802.295831] LNet: Removed LNI 192.168.206.134@tcp32 [14802.298960] LNet: Removed LNI 192.168.206.134@tcp33 [14802.302457] LNet: Removed LNI 192.168.206.134@tcp34 [14802.308799] LNet: Removed LNI 192.168.206.134@tcp35 [14802.311718] LNet: Removed LNI 192.168.206.134@tcp36 [14802.314994] LNet: Removed LNI 192.168.206.134@tcp37 [14802.318393] LNet: Removed LNI 192.168.206.134@tcp38 [14802.320800] LNet: Removed LNI 192.168.206.134@tcp39 [14802.329351] LNet: Removed LNI 192.168.206.134@tcp40 [14802.332690] LNet: Removed LNI 192.168.206.134@tcp41 [14802.335170] LNet: Removed LNI 192.168.206.134@tcp42 [14802.345849] LNet: Removed LNI 192.168.206.134@tcp43 [14802.349658] LNet: Removed LNI 192.168.206.134@tcp44 [14802.359986] LNet: Removed LNI 192.168.206.134@tcp45 [14802.370881] LNet: Removed LNI 192.168.206.134@tcp46 [14802.372832] LNet: Removed LNI 192.168.206.134@tcp47 [14802.380312] LNet: Removed LNI 192.168.206.134@tcp48 [14802.385590] LNet: Removed LNI 192.168.206.134@tcp49 [14802.407497] LNet: Removed LNI 192.168.206.134@tcp50 [14803.422031] Key type .llcrypt unregistered [14803.426611] Key type ._llcrypt unregistered [14828.329725] Key type ._llcrypt registered [14828.331419] Key type .llcrypt registered [14828.469419] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [14845.290255] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [14846.373064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14846.574083] alg: No test for adler32 (adler32-zlib) [14847.808743] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [14848.148703] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [14849.888479] Key type lgssc registered [14851.548322] Lustre: Echo OBD driver; http://www.lustre.org/ [14890.638526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14890.671941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14892.012332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14892.047296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14892.170750] Lustre: lustre-MDT0000: new disk, initializing [14892.320915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14892.348935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14897.197173] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [14909.305809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14909.408310] Lustre: 505808:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14909.474282] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14909.477881] Lustre: Skipped 1 previous similar message [14909.546481] Lustre: lustre-MDT0001: new disk, initializing [14909.628294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14909.660702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14909.670031] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14915.157436] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [14925.732990] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14932.589904] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14941.812092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14942.082917] Lustre: *** cfs_fail_loc=90f, val=0*** [14942.224714] Lustre: lustre-OST0000: new disk, initializing [14942.232739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14942.238631] Lustre: 507828:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14942.372548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14948.997891] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [14952.693488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14952.708355] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14952.889927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14960.333475] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14973.420076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14973.423021] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14973.426290] LustreError: Skipped 1 previous similar message [14973.431355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14973.450724] Lustre: Skipped 1 previous similar message [14973.473298] Lustre: Skipped 1 previous similar message [14978.530100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14978.551889] Lustre: Skipped 1 previous similar message [14983.136133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14983.367781] Lustre: server umount lustre-OST0000 complete [14988.770531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14988.793566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14993.753268] Lustre: server umount lustre-MDT0000 complete [14998.338681] LustreError: 505802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788483398 with bad export cookie 5598477370568191336 [14998.345308] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14998.359861] LustreError: 505802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14998.835285] Lustre: server umount lustre-MDT0001 complete [15011.860211] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [15015.467987] Key type lgssc unregistered [15015.697788] LNet: 509737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15015.724118] LNetError: 509737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15016.808589] LNet: Removed LNI 192.168.206.134@tcp [15017.840712] Key type .llcrypt unregistered [15017.849134] Key type ._llcrypt unregistered [15039.357157] Key type ._llcrypt registered [15039.358385] Key type .llcrypt registered [15039.498021] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [15040.764543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15040.810187] alg: No test for adler32 (adler32-zlib) [15042.133512] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [15042.429227] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [15044.104276] Key type lgssc registered [15045.454967] Lustre: Echo OBD driver; http://www.lustre.org/ [15055.436571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15055.483569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15057.303641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15063.393771] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [15066.593240] LustreError: 511002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15066.637708] LustreError: 511002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15071.718938] LustreError: 511003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15073.357943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15074.068973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15079.518720] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [15088.063409] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15093.814989] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15100.110041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15100.441316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15106.654685] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [15113.830387] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15128.664744] loop0: detected capacity change from 0 to 204800000 [15128.812369] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15128.834655] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15128.865388] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15135.324891] loop0: detected capacity change from 0 to 204800000 [15135.473132] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15135.487288] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15135.512765] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15140.312220] loop0: detected capacity change from 0 to 204800000 [15145.891929] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15146.014843] Lustre: fs15246-OST0000: new disk, initializing [15146.018062] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15146.023618] Lustre: 514595:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15146.079779] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15152.748573] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [15163.452185] loop1: detected capacity change from 0 to 204800000 [15170.118826] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15170.204689] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15170.352978] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15170.576533] Lustre: fs15246-MDT0000: new disk, initializing [15170.787042] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15170.827436] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15171.018017] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15172.072986] Lustre: 515718:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000173b6937 x1875360814777984/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788483588 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15177.657112] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [15181.180124] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15181.198297] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15181.266355] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15208.448830] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15208.468929] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15213.551230] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15214.603379] Lustre: server umount fs15246-MDT0000 complete [15218.669078] LustreError: 512843:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15223.789318] LustreError: 510998:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15224.258942] Lustre: server umount fs15246-OST0000 complete [15239.141991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15239.147938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15239.165354] Lustre: Skipped 1 previous similar message [15239.189076] Lustre: Skipped 1 previous similar message [15244.267376] LustreError: 512822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15244.280090] LustreError: 512822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15244.393319] Lustre: server umount lustre-OST0000 complete [15249.379404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15249.382200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15249.403013] Lustre: Skipped 1 previous similar message [15249.427323] Lustre: Skipped 1 previous similar message [15254.499253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15254.511834] Lustre: Skipped 1 previous similar message [15255.838131] Lustre: server umount lustre-MDT0000 complete [15259.617793] LustreError: 512843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15259.642494] LustreError: 512843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15260.647229] LustreError: 510984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788483661 with bad export cookie 902695722137433875 [15260.662167] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15260.681334] LustreError: 510984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15261.613256] Lustre: server umount lustre-MDT0001 complete [15274.450716] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [15277.960567] Key type lgssc unregistered [15278.398650] LNet: 518829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15278.412392] LNetError: 518829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15278.425934] LNet: Removed LNI 192.168.206.134@tcp [15279.397650] Key type .llcrypt unregistered [15279.400799] Key type ._llcrypt unregistered [15300.593271] Key type ._llcrypt registered [15300.595315] Key type .llcrypt registered [15300.696798] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [15301.772989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15301.938501] alg: No test for adler32 (adler32-zlib) [15302.928838] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [15303.139137] LNet: Added LNI 192.168.206.134@tcp [8/256/0/180] [15304.777059] Key type lgssc registered [15306.108149] Lustre: Echo OBD driver; http://www.lustre.org/ [15348.545575] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [15360.783988] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [15400.414627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15400.456674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15401.804221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15401.839436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15401.940538] Lustre: lustre-MDT0000: new disk, initializing [15402.086848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15402.118407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15406.797784] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [15419.274409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15419.400926] Lustre: 523300:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15419.576479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15419.585681] Lustre: Skipped 1 previous similar message [15419.734341] Lustre: lustre-MDT0001: new disk, initializing [15419.910519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15419.973499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15419.997642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15426.243986] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [15438.364382] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15445.491255] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15453.310426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15453.703617] Lustre: lustre-OST0000: new disk, initializing [15453.707503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15453.714694] Lustre: 525266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15453.855461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15460.110460] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_default_debug -1 all [15461.556480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15461.578926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15461.727309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15469.834227] Lustre: DEBUG MARKER: oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15475.032528] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15475.223379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15480.265436] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15480.563211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15485.428858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15485.452161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15485.461093] Lustre: Skipped 1 previous similar message [15490.533932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15490.542490] Lustre: Skipped 1 previous similar message [15495.653939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15495.667771] Lustre: Skipped 1 previous similar message [15497.184243] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15497.621080] Lustre: server umount lustre-OST0000 complete [15505.890433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15505.896975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15505.902666] Lustre: Skipped 2 previous similar messages [15505.911994] Lustre: Skipped 1 previous similar message [15511.335578] Lustre: server umount lustre-MDT0000 complete [15515.634044] LustreError: 523291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788483916 with bad export cookie 17062324744321532464 [15515.638809] LustreError: MGC192.168.206.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15515.652284] LustreError: 523291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15516.186385] Lustre: server umount lustre-MDT0001 complete [15543.682585] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing set_hostid [15554.187656] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing load_modules_local [15585.996959] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15313 sec ========= 21:06:24 (1788483984) [15588.086666] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:06:26 (1788483986) === [15592.245644] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:06:30 (1788483990) === [15638.762800] Lustre: DEBUG MARKER: oleg634-server.virtnet: executing unload_modules_local [15641.834748] Key type lgssc unregistered [15642.109636] LNet: 531916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15642.119391] LNetError: 531916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15642.135545] LNet: Removed LNI 192.168.206.134@tcp [15643.139277] Key type .llcrypt unregistered [15643.152705] Key type ._llcrypt unregistered